Assessment task 2: Planning, implementation and evaluation of a non-communicable disease prevention initiative
This assignment uses a suburban state primary school as a setting for the prevention of overweight and obesity. The task involves an extensive whole-of-school initiative that targets specific stakeholder groups usually associated with a school (pupils, teachers, families, community). Students will work as Public Health Officers working for the state Department of Health on the planning, implementation and evaluation of this intervention. Students will identify how this new initiative will be deployed, what resources, support and incentives will be required and what concerns and/or barriers the initiative may encounter. Students will also develop information for the stakeholder groups and other communication strategies, and outline an evaluation strategy. Importantly, a direct link between the intervention design and risk reduction in the target group must be demonstrated.
Submission will comprise the following (approximate distributions of word count):
Demographic characteristics of the specific school and surrounding community (600 words)
Etiology and epidemiology of overweight and obesity in the target group (700 words)
Justification of stakeholder group selection (200 words)
Description of the intervention (700 words)
Outline of the communication strategy (300 words)
Outline of the evaluation framework (500 words)
Evaluation framework for health promotion and disease prevention programs Evaluation framework for health promotion and disease prevention programs Evaluation framework for health promotion and disease prevention programs Evaluation framework for health promotion and disease prevention programs If you would like to receive this publication in an accessible format, please phone +61 3 9096 0393 using the National Relay Service 13 3677 if required, or email:
[email protected]. This document is also available in PDF format on the internet at: www.health.vic.gov.au/healthpromotion/evidence_evaluation/cdp_tools.htm Published by the Prevention and Population Health Branch, Victorian Government Department of Health, Melbourne, Victoria © Copyright, State of Victoria, Department of Health, 2010 This publication is copyright, no part may be reproduced by any process except in accordance with the provisions of the Copyright Act 1968. Authorised by the State Government of Victoria, 50 Lonsdale Street, Melbourne. Printed on sustainable paper by Impact Digital, Unit 3-4, 306 Albert St, Brunswick 3056. December 2010 (1102008) Suggested citation: Prevention and Population Health Branch 2010, Evaluation framework for health promotion and disease prevention programs, Melbourne, Victorian Government Department of Health. Contents Introduction 1 Aim of the framework 1 Complementary activities 1 Link with the Integrated Health Promotion Resource Kit 1 Limitations 1 Why develop an evaluation plan? 2 Step 1. Describe the program 3 Program logic 3 Why should you use program logic? 3 Program logic outline 3 Health inequalities 4 Step 2. Evaluation preview 5 Engage stakeholders 5 Clarify the purpose of the evaluation 5 Identify key evaluation questions 5 Identify resources for the evaluation 5 Step 3. Focus the evaluation design 7 Study design 7 Data collection tools 7 Data collection methods 7 Process evaluation 8 Impact/outcome evaluation 10 Health inequalities 12 Step 4. Collect data: coordinate the data collection 13 Maximising response rates 13 Step 5. Analyse and interpret data 13 Step 6. Disseminate the lessons learned 14 Dissemination strategies 14 Business case for new and continuing data collections 15 References 16 Evaluation framework for health promotion and disease prevention programs 1 Aim of the framework This framework aims to improve the evaluation of health promotion and disease prevention programs by: • providing guidance on how to write an evaluation plan (included in this document) • including an example of a good evaluation plan (through the Health Promoting Communities: Being Active and Eating Well (HPC:BAEW) evaluation plan) • specifying some agreed parameters for good evaluation, for example, identifying a good study design for impact evaluation (included in this document) • specifying an agreed list of indicators (‘the indicators’), which allows comparison of the impacts and outcomes of different programs. Complementary activities This framework should be complemented by the following actions: • evaluation plans that are developed jointly by program staff, key stakeholders and staff with evaluation or research expertise • a commitment from management and staff to support quality evaluation, so that evaluation plans are written simultaneously with program plans and before program implementation or tendering • a commitment from management and staff to use the results of evaluations in future program design. Link with the Integrated Health Promotion Resource Kit The language used in this document is consistent with the language of ‘integrated health promotion’ as used in the resource kit (Department of Human Services 2003a) and evaluation guides (Department of Human Services 2003b, Round et al. 2005). However, not all sectors employ this language, and different sectors may apply different terminologies. To increase understanding, defi nitions and explanations are given throughout this document where differences in use may arise. This framework is designed to complement the integrated health promotion evaluation resources (Department of Human Services 2003b, Round et al. 2005), and readers are referred to these resources for further details. Limitations This framework may not be appropriate in all cases, and fl exibility is required when writing evaluation plans that must meet the evaluation requirements of specifi c programs. Large-scale evaluations are not required for all programs, and it may be appropriate to simplify the evaluation plan in terms of number of questions asked, range of indicators measured and complexity of study design. (See the section ‘Identify resources for the evaluation’ in Step 2, and Step 3 for more information.) Introduction 2 Evaluation framework for health promotion and disease prevention programs Defi nition The terms ‘evaluation plan’ and ‘evaluation framework’ are often used interchangeably. An evaluation plan should be developed for all new programs before they are implemented. The evaluation plan should be written alongside the overall program plan. It should allow you to: • identify the objectives of the evaluation • clarify roles and responsibilities of those involved in the evaluation • determine the most appropriate evaluation strategy/design • clarify assumptions/evidence on which program design and implementation were based • outline how a program intends to produce results • design the most appropriate evaluation questions to measure the impact of the objectives • determine the most appropriate data collection methods • outline how the evaluation results will be disseminated • cost the evaluation. The six steps outlined in this guide are based on the framework used in the Planning for effective health promotion evaluation resource (Round et al. 2005), with some modifi cations. Flexibility can be exercised in the headings used, provided the key issues/parameters covered here are included. The six steps in the evaluation framework for health promotion and disease prevention programs are: 1. Describe the program 2. Evaluation preview 3. Focus the evaluation design 4. Data collection 5. Data analysis and interpretation 6. Disseminate lessons learned Why develop an evaluation plan? 3 This section should briefl y outline what the program is, including its goals and objectives, target groups, the policy context, supporting evidence and key assumptions. These issues should have been addressed as part of your program planning, and you should summarise the detail here to focus the evaluation. The links between program planning and evaluation: • goal is measured by outcome evaluation • objective is measured by impact evaluation • how well your interventions/activities/ strategies are implemented is measured by process evaluation. The description of the program should also include the ‘program logic’ and consider how the program is addressing health inequalities. Program logic Defi nition The term ‘program logic’ is frequently used interchangeably with the terms ‘program theory’, ‘logic model’ and ‘causal model’. Why should you use program logic? Using a program logic model in the program planning and evaluation planning stages can assist you to identify the activities, impacts and outcomes that need to be evaluated. Logic models can also provide a theoretical framework for your program design when evidence is less robust. Models such as this can be developed for smaller components or objectives of the program, or they can be used to represent all programs across a community or state (US Department of Health and Human Services 2002). See Figure 1 Program logic: Underlying intention of the Health Promoting Communities: Being Active and Eating Well initiatives as an example program logic model that is applicable to a whole-of-community health promotion program for nutrition and physical activity. Program logic outline Program logic models can also include a column on the left to identify inputs, or the resources needed to operate the program (this is not included in this example). See Figure 1 Program logic: Underlying intention of the Health Promoting Communities: Being Active and Eating Well initiatives. Models can also include a column after activities to defi ne outputs, or the types, levels and targets of services to be delivered by the program (these are not included here, but can be found in Table 2 Some example key activities, outputs and reach indicators—for process evaluation). Outputs link to your process evaluation indicators. Impacts and outcomes should link to your impact and outcome indicators. See Table 3 Example impact and outcome indicators for nutrition, physical activity and obesity programs for an example. Defi nition Different defi nitions for impacts and outcomes are evident in the evaluation literature. Here, we defi ne impact as the intermediate effect that health promotion programs have on populations, individuals or their environments (Round et al. 2005). Outcome is defi ned as the long-term effect of programs and may include reductions in incidence or prevalence of health conditions, changes in mortality, sustained behaviour change, or improvements in quality of life (Round et al. 2005). Step 1. Describe the program 4 Evaluation framework for health promotion and disease prevention programs The second column of program activities is based on the language used in the Integrated Health Promotion Resource Kit (Department of Human Services 2003a). The design of your program logic and language used is fl exible. The important point for evaluation purposes is that your inputs and activities match the expected impacts and outcomes. For example, you may expect an impact on physical activity, but none of your activities is actually aimed at changing physical activity levels. This should highlight the need to reconsider your activities or change your impacts. Other examples of, and alternative approaches to, program logic models can be found in the W.K. Kellogg Foundation Evaluation handbook (1998) and Logic model development guide (2004). Figure 1 Program logic: Underlying intention of the Health Promoting Communities: Being Active and Eating Well initiatives Health inequalities Regarding health inequalities, the program must be clear about how population-wide approaches are used to reduce unequal health outcomes and to ensure that they do not inadvertently widen inequalities. Targeted interventions are often used in conjunction with population-wide approaches to minimise this risk and further equality goals (Boyd 2008). 5 Step 2 involves these components: • engaging stakeholders • clarifying the purpose of the evaluation • identifying key questions • identifying resources for the evaluation. This section should identify the key stakeholders involved; clarify the aspects of the program that are to be evaluated and the purpose of the evaluation, including who will use the results and how, for example, to determine future funding. Engage stakeholders More information on this can be found in the Planning for effective health promotion evaluation resource by Round et al. (2005: p. 9). List your key stakeholders here and consider including them in your evaluation planning, for example, by asking them to help in constructing the program logic. Clarify the purpose of the evaluation More information on this can