In this project, you will conduct your own research and then create and evaluate a health promotion program that targets one of the following issues: tobacco use cessation, type 2

HCA 3306, Community Health 1

Course Learning Outcomes for Unit VII Upon completion of this unit, students should be able to:

2. Recognize effective principles of health programming for community health on a global scale. 2.1 Indicate the steps involved in organizing a community health program. 2.2 Define the steps needed for effective evaluation of community health programs.

7. Assess methods for improving health care delivery in the United States.

7.1 Describe health care performance improvement approaches and methods Required Unit Resources Chapter 15: Systems Thinking and Leadership in Community and Public Health Unit Lesson

Centers for Disease Control and Prevention Framework for Program Evaluation In community health, much of our time and effort is invested in designing and implementing programs for community health interventions. The goal of these programs is to address pressing health issues in the community and to improve individual health and public health as a result. As community health leaders, we have a responsibility to continuously and proactively seek better health for our clients and for the community overall. But how do we know if our programs are really working? How do we know if we are achieving our desired goals and truly enhancing the health of the communities we serve? Well, that question brings up the topic of a framework for program evaluation, and the question takes us to the United States Centers for Disease Control and Prevention (CDC) in Atlanta. The CDC is truly the world’s leader in program evaluation thinking and applications. Public health organizations all over the world utilize the CDC model for evaluating its programs. It can accurately be said that Americans have shown leadership and innovation in public health evaluation. In this lesson, we will explain what program evaluation is, why it is so important, and the steps necessary for planning and conducting evaluations. We will also discuss the standards that currently exist for evaluating health care programs of all kinds. A key aspect of this lesson is the concept of stakeholders and ways to engage them in planning and program evaluation. Let’s start with a big picture view of evaluation. We spend a lot of time, effort, and money creating programs to improve the health of our communities. But is that time, effort, and money truly making a positive difference? Evaluation sets out to answer that very important question.

Evaluation There are several ways to think about evaluation of health programs. Evaluation is systematically evaluating the merit of a health program and its worth in terms of improving community health, including assessment of program operations and outcomes using a pre-established set of criteria for assessing the program. Evaluation includes determining the value of a program or activity and what it adds to community health. This is the value-added piece. Evaluation is related to research, but it is not the same thing as research. Let’s take a minute here to understand the difference. Research focuses on producing generalizable knowledge that can be used to

UNIT VII STUDY GUIDE Leadership in Public Health

HCA 3306, Community Health 2

UNIT x STUDY GUIDE Title

apply our knowledge from a sample to a larger population. It starts with research-derived questions and a paradigm stance of what we commonly term a hypothesis. Research is done in a very controlled setting, and the intention is to publish results for the professional community. Meanwhile evaluation produces knowledge that is intended for immediate use in the program. It starts with program-generated or funder-generated questions, and it takes place in an action setting that is much less controlled than the research setting. Evaluation is typically not published, but particularly interesting program evaluations are sometimes shared in the professional literature. Research sets out to prove something; whereas, evaluation sets out to improve something. That is a great and simple way to look at the difference. Evaluation is all about making a positive difference in the program under consideration. In that sense, it is similar to accreditation visits for health care facilities and academic programs. Many readers will have experienced accreditation visits to their own medical facilities and will have some understanding of this connection. Surveillance and monitoring are aspects of evaluation. Surveillance looks at disease or risk behaviors over time, while monitoring looks at changes in program outcomes over time. But evaluation does not just look at these behaviors and outcomes, it asks why these changes are taking place.

What Type of Programs Can Be Evaluated? Just about any type of program can be evaluated:

• direct health interventions, • community mobilization programs and services, • surveillance systems, • monitoring systems, • health policy initiatives, • disease outbreak investigations, • clinical laboratory testing, • communication and education campaigns, • programs to create infrastructure, • community health education programs, • community-wide or focused screening programs, and • administrative effectiveness.

These are just a few examples. Truly any community health program needs evaluation and should be evaluated.

When Should We Evaluate? Evaluation has a role at all phases in the program planning and delivery process, from conception to completion of the program. We can assess a developing program to see if it is headed in the right direction. We can assess a stable, mature health program to see how well it is meeting its objectives and community needs, and we can evaluate a program after it has ended. Some health programs are not ongoing and exist for a definite specified time period (3–5 years being common), and then they are terminated. Evaluation can help us to decide if time-limited health programs actually worked and whether they should be repeated or not.

Why Do We Evaluate Programs? Evaluation helps us to gain insight into our programs and their operations, and it looks at their direction and their effectiveness. It helps us to understand what works and what does not work. We can improve practices for better health impact through evaluation. Evaluation is all about modifying current practices so that they work better and achieve more. Evaluation measures progress toward stated objectives and goals. Sometimes, evaluation helps us document for our funders that their money is well spent, and that can lead to expansion and increased funding. It can help us to demonstrate accountability.

HCA 3306, Community Health 3

UNIT x STUDY GUIDE Title

CDC Steps in Community Health Program Evaluation The CDC has developed a six-step process for program evaluation, which has become the clear standard for evaluating health care programs of all kinds:

• engage stakeholders, • describe the program, • focus the evaluation design, • gather credible evidence, • justify conclusion, and • ensure use and share lessons learned (CDC, n.d.).

This approach emerged over time because evaluation is only valuable if the findings are actually used to make a positive difference. Early involvement of stakeholders is crucial here because there must be a market for the product of the evaluation. Stakeholders help us to ask the most important and most relevant questions so that the results of the evaluation will actually be meaningful and utilized. There is nothing worse than conducting an extensive evaluation process and seeing the resulting document go onto a shelf where it is never utilized. The CDC has tried very hard over the years to not let that happen. A real key is finding out who cares about the success of the program, the stakeholders, and very importantly, we let those stakeholders help us to define the success of the program. Stakeholders are crucial here. Do they think about the program in the same way that we do as evaluators? Often, evaluators learn from the stakeholders that certain things are more important.

Four Standards for Effective Evaluation So what standards will evaluators utilize as they move through the CDC model? Four standards have consistently been most helpful, and they should be considered in every evaluation (CDC, n.d.).

• Utility refers to who actually needs the information from the evaluation and what they will do with it. Clearly, the evaluation should be conducted in a way that meets stakeholder and funder needs and generates information that is helpful. A key question under utility is whether or not the evaluation provides relevant and useful information in a timely manner.

• Feasibility refers to how much money and time can we realistically put into the evaluation process. There may be a lot of things that we would like to include in the evaluation, but there are real-world time and budget constraints. We need to pick the things that are most important and can realistically be evaluated. A key question under feasibility is whether or not the planned evaluation steps are realistic given the time resources and expertise of the evaluators at hand.

• Propriety refers to how we need to take steps to conduct the evaluation ethically and in upmost respect for the privacy of medical information that will be included in the evaluation. The Health Insurance Portability and Accountability Act provides important guidance here, and we must carefully follow its guidelines. A key question is whether or not the evaluation protects the rights of individuals and protects the welfare of clients.

• Accuracy refers to how we need to design our evaluation in such a way that the information obtained is accurate and reliable for decision-making. A key question is whether or not the evaluation provides findings that are valid, given the needs of those who will utilize the results.

Conclusion

Hopefully, this lesson has provided some insights into the process of evaluating community health programs. The process is important. It has been fine-tuned over the years under CDC leadership as evaluators have learned more and more about what works well and what does not. The completed evaluation is shared with all stakeholders, and, if appropriate, it is shared with local media. Much is riding on the results of the evaluation process. The continuation of the program definitely depends upon evaluation results. Continued funding is often at risk, and opportunities to replicate the program in other communities often hinge upon evaluation results for a particular program. So a good bit of time and thought goes into the design of the evaluation.

HCA 3306, Community Health 4

UNIT x STUDY GUIDE Title

Also, the evaluators must be objective in their assessments, which is so important. The best evaluations are done by community health experts from outside the community and by people who have no vested interest in the program. Objectivity is best in that scenario.

Reference Centers for Disease Control and Prevention. (n.d.). A framework for program evaluation. U.S. Department of

Health and Human Services. https://www.cdc.gov/eval/framework/index.htm Suggested Unit Resources In order to access the following resource, click the link below. The following web page will help you understand the Racial and Ethnic Approaches to Community Health (REACH) program. The article will cover the impact of the program across REACH communities. Centers for Disease Control and Prevention. (2020). REACH program impact. U.S. Department of Health and

Human Services. https://www.cdc.gov/nccdphp/dnpao/state-local- programs/reach/program_impact/index.htm