Can a community arts event actually get people to do more exercise? Does a digital app which encourages you to dance at random hours of the day improve your sense of social connection? If you run around with a story and a list of clues, do you develop a greater sense of belonging in your town? This article summarises some research we did in the context of arts for health promotion to develop a framework for evaluating the impact of VicHealth's active arts strategy. I am sharing it because it might help a little with your thinking if you are trying to work out how to find out if your arts activity actually changes the way people behave, without the benefit of a randomised controlled trial (research nerd joke ;-). If anyone has further insights to share, please do!
The social model of health
Health promotion activities take place within the context of the social model of health attempts to address broader influences on health, such as social, cultural, environmental and economic factors, rather than respond directly to disease and injury. It focuses on policies, education and health promotion to effect social change to provide the prerequisites for health.
Social models of health attempt to:
Change the broader determinants of health e.g. gender, socioeconomics, built environment, social inequities
Empower individuals and communities with skills, knowledge and self-efficacy to make positive health decisions
Access to health care, addressing physical and cultural barriers e.g education, location, language, culture
Collaborate across departments and stakeholders
Health promotion models and theories of change
Health promotion models and theories of change vary in terms of their emphasis. Earlier theories tended to focus on personal decision-making behaviours, whilst later theories take into account personality factors, self-efficacy, intrinsic and extrinsic motivations to change. Broadly speaking, health promotion models and theories of change can be grouped as follows:
Behavioural change theories
Ecological theories and models
Planning models
Communication theories
Nursing theories of health promotion and change (Raingruber, 2014).
How health promotion theories can inform arts intervention theories of change
When we were developing the VicHealth Active Arts Strategy theory of change, we identified the following as particularly helpful and apposite health promotion theories:
Social cognitive theory
Self-determination theory
Salutogenic theory
Diffusion of innovations theory
Social cognitive theory(SCT) is based on vicarious learning. Behaviour is learned via observation, imitation, positive reinforcement and noticing the benefits to others. SCT emphasises self-efficacy, which refers to a person’s sense of confidence to act (Raingruber, 2014: 58-9).
Self-determination theory focuses on intrinsic motivations for action. People who are intrinsically motivated are interested and satisfied by doing the activity, rather than focusing on an extrinsic reward such as recognition or income.
Intrinsic motivation can flourish when a person has a sense of connection with others, a sense of agency in their own lives and a feeling of competence to deal with one’s environment and produce positive outcomes (Raingruber, 2014: 59).
Salutogenic theory focuses on building people’s ‘strong sense of coherence,’ which results in seeing the world as meaningful and manageable. Coherence can be developed through positive relationships and meaningful pursuits (Raingruber, 2014: 67).
Diffusion of innovations theory describes the stages of change as: knowledge, persuasion, decision, implementation and confirmation. A health promotion program would be adopted if it could demonstrate that it is: better than other options; compatible with existing values and needs; trialable; producing observable results; easy to use, understand and communicate; adoptable with a minimal investment of time and risk; and usable with a moderate level of commitment. Roles in the adoption of innovations (including innovative health approaches) include opinion leaders, change agents and change aides.
Approaches to the Evaluation of Health Promotion Impacts
There are a number of different approaches to evaluating the impacts of arts-based (and other) health promotion activities. The key factors in choosing an approach are:
What is the purpose of the evaluation?
What are our resources the evaluation?
Are the causal links between proximal indicators to distal outcomes already established?
Three of the main evaluative approaches which we tend to adapt and draw from when measuring the social and behavioural impact of an arts program include the following:
Incremental evidence evaluation (commonly known as a program logic or impact evaluation)
PRECEDE/PROCEED model
RE-AIM framework
Incremental evidence evaluation (program logic / impact evaluation)
In this approach, researchers gather evidence about the effectiveness of an health promotion program or intervention in its real world setting, effecting impact on a behavioural determinant of health outcomes (Nutbeam, 1998). It can be understood as a program logic evaluation, or impact evaluation, where the desired impacts of the program are also determinants of positive health outcomes (Thorogood and Coombes, 2010: 12).
This approach allows researchers to evaluate the effectiveness of programs happening in the real world, rather than focusing on expensive, experimental situations such as random-controlled trials. Change occurs on a continuum. If links are already well-established, then you do not have to prove causal relationships again. Instead, you can focus on proximal indicators which are known to be related to distal outcomes (Green and Tones: 1999).
In deciding what level of outcome to measure, researchers consider how the evaluation will be used. The evaluation can occur at various levels of complexity, as follows:
Process evaluation – how was the program carried out?
Adequacy – did the expected changes occur?
Plausibility – were the outcomes actually due to the intervention vs external factors?
Probability – did the intervention have a (health) effect?
PRECEDE-PROCEED Model (PPM)[1]
The PPM model aims to describe proximal, intermediate and distal outcomes of health promotion programs. The approach is based on the assumption that interventions will be effective if they:
Come from the community
Are well planned
Are based on data
Are seen by the community as feasible
Include multiple strategies woven together
Rely on feedback and progress evaluation (Green and Kreuter, 1992)
The PRECEDE part of the model focuses of educational factors that influence change, whilst the PROCEED part looks at the importance of ecological factors.
The model involves the following steps:
Social assessment: community members identify their own health promotion needs
Epidemiological assessment: Identify the health problems of the community using national statistics
Behavioural and environmental assessment: Identify factors contributing to the problem. Factors are ranked according to importance and feasibility of changing them. Most changeable and most important factors are priority targets.
Educational and ecological assessment: Identify predisposing, reinforcing and enabling factors for the desired health behaviours, and develop measurable objectives.
Administrative and policy assessment: Design interventions and the resources, organisational changes and circumstances required for success.
Process evaluation: Evaluating the implementation of the intervention.
Impact evaluation: Evaluating the impact of the intervention on the reinforcing, predisposing and enabling factors, behaviours, lifestyle and environment.
Outcome evaluation: Evaluation of health outcomes.
The RE-AIM Framework[2]
This framework is designed as a planning tool to select between health promotion projects for investment. Projects are assessed according to five dimensions:
Reach: how many people will be influenced by the program
Efficacy/effectiveness: produces positive outcomes with few unintended consequences
Adoption: participation rate and representativeness
Implementation: process evaluation as to whether the program was implemented as intended
Maintenance: long-term utilisation of the given health behaviour, and whether a program is sustainable even if resources change.
[1]PRECEDE stands for ‘predisposing, reinforcing and enabling constructs in educational/environmental diagnosis and evaluation.’ PROCEED stands for ‘policy, regulatory, and organisational constructs in education and environmental development.’
[2] RE-AIM stands for reach, efficacy/effectiveness, adoption, implementation and maintenance (Glasgow, Vogt and Boles, 1999).
In conclusion
Most evaluations of arts interventions happen on limited budgets and timeframes. You may not have a large sample population and a large budget, but don't despair! Evaluation is still possible and worthwhile, because you may be able to evaluate whether your project is contributing to the determinants of behavioural change. And that is the start of the answer.
References
Bailey, Jackie and Yang, Hung-Yen. (2014). Play Me I’m Yours: Evaluation Report and Appendices. Arts Centre Melbourne: Melbourne.
Foreman-Wernet, L and Dervin. B. (1011). ‘Cultural Experience in Context: Sense-Making in the Arts.’ The Journal of Arts Management, Law and Society 41: 1-37.
Gilmour, J, MacDowall, L and Oliver, J. (2013). The Arts, Physicality and Connection: An Evaluation of VicHealth’s MOTION Program. VicHealth: Melbourne.
Glasgow, R, Vogt, T and Boles, S. (2011). ‘Evaluating the public health impact of health promotion interventions: the RE-AIM framework.’ Social Science and Medicine 73: 383-390.
Green, J and Tone, K. (1999). ‘Towards a secure evidence base for health promotion.’ Journal of Public Health Medicine 21(2): 133-9.
Green, L and Kreuter, M. (1992). ‘CDC’s planned approach to community health as an application of PRECEDE and an inspiration for PROCEED.’ Journal of Health Education 23(3): 124-147.
Kelaher, M, et al. (2014). ‘Evaluating community outcomes of participation in community arts: a case for civic dialogue.’ Journal of Sociology 50(2): 132-149.
Loy, C, et al. (2013) ‘Crowd Counting and Profiling: Methodology and Evaluation.’ In Modeling, Simulation, and Visual Analysis of Large Crowds. Ali, S et al. (Eds). Springer: New York. 347-382.
McCarthy, Kevin, et al (2004). Gifts of the Muse: Reframing the Debate about the Benefits of the Arts. RAND Corporation: New York.
Nutbeam, D. (1998). ‘Evaluating health promotion – progress, problems and solutions.’ Health Promotion International 13: 27-43.
Radbourne, J, et al. (2009). ‘The Audience Experience: Measuring Quality in the Performing Arts.’ International Journal of Arts Management 11(3): 16-29.
Raingruber, Bonnie. (2014). Contemporary Health Promotion in Nursing Practice. Jones & Bartlett Publishers: Burlington MA.
Sherwood, N and Jeffery, R. (2000). ‘The Behavioural Determinants of Exercise: Implications for Physical Activity Interventions.’ Annual Review of Nutrition 20: 21-44.
Synergistiq. (2014). Evaluation of MOTION: Final Report. VicHealth: Melbourne.
Thorogood, Margaret and Coombes, Yolande. (2010). Evaluating Health Promotion: Practice and Methods. Oxford University Press: Oxford.
Victorian Government Department of Human Services. (2003). Measuring Health Promotion Impacts: A Guide to Impact Evaluation in Integrated Health Promotion. Department of Human Services: Melbourne.
Watson, R and Yip, P. (2011). ‘How many were there when it mattered? Estimating the sizes of crowds,’ Significance, September, 104-107.