Consider the program SNAP that you are improving and draft a memo to your supervisor or board. (https://www.fns.usda.gov/snap/supplemental-nutrit

1122924 – SAGE Publications, Inc. (US) ©

articulate, but they influence the way in which we view the world and the choices we make. As with any field, there are particular issues that help to define our ideology. I will refer to these issues as calibrators, taken from multiple definitions of the term. Calibrators divide or mark a scale with gradations to determine the degree of something along that scale. Likewise, calibrators can also be plans that have a specific use or application.

Ideology: a system of beliefs that we use to explain and develop solutions; a philosophy or a way of thinking about a certain topic or issue.

A calibrator, used in the context of evaluation, is a continuum upon which we can consider where our beliefs fall and through which we can apply those beliefs in a particular context. The calibrators presented in this section are not meant to be dichotomous, that is, either-or, but rather a range with the extremes presented as a way to consider where your beliefs might fall along the continuum, as well as the strength with which you hold that belief. Three areas of calibrators are discussed: design calibrators, role calibrators, and methods calibrators. These categories and the calibrators within them are not mutually exclusive, as you will notice some similarities. But, taken as a whole, they should provide you with considerations to shape and focus your own ideology.

Calibrator: a continuum upon which we can consider where our beliefs fall and through which we can apply those beliefs in a particular context.

4.2.1 Design Calibrators

Design calibrators help us to organize our thinking around the overarching research design used in evaluation. Evaluation design is covered in detail in Chapter 8, though considerations regarding how you make design choices will be presented below. It should be noted, however, that other factors often determine what evaluation design we can and cannot use in a specific situation. Thus, regardless of our thinking about these calibrators, it does not necessarily mean that we will be in a position to use any design we choose or make unilateral decisions regarding evaluation design. The design we use will be influenced by the resources, both people and financial, available to the evaluation and the context in which the evaluation is implemented. For instance, in some contexts, the environment might facilitate or even promote stakeholder involvement and foster evaluator access to program participants. In other contexts, the environment might present barriers to evaluation.

Calibrator D1: Design Structure.

At one extreme on the design structure calibrator is the medical model of research. At the other extreme is the anthropological model of research. Important questions include the following:

To what extent can and should evaluation be conducted in a controlled environment, using research designs closely aligned to the medical model of research?

To what extent can and should evaluation be conducted in natural settings, much like research designs used in anthropological research?

In what ways is there a trade-off between causation in controlled settings versus correlation in natural settings?

What is the ideal design structure for program evaluation? To what extent does this design structure vary by the purpose of the evaluation being primarily formative or primarily summative?

Some evaluators believe that unless the underlying research design is of sufficient rigor to make causal conclusions, there is little value in utilizing resources to conduct the evaluation. Other evaluators believe that the controls necessary to implement an experiment based on the medical model create an unrealistic environment within which to evaluate the program, thus limiting the generalizability of findings. There is no doubt that veteran evaluators have already formed ideologies in this area and have strong preferences for design structure in various evaluation environments. Honestly, I understand and respect the arguments at both extremes, and I hesitate to share my ideologies for fear of influencing your own deliberations. However, I will say two things: If the ability to relate a program’s strategies to its goals is compromised, generalizability is a moot issue. Likewise, if the effort for program staff to conduct an evaluation is so cumbersome due to environmental changes necessary to create a controlled environment, evaluation is less likely to occur. Oh—and one more thing—making decisions based on some information is better than decisions made without data (either because evaluation is too cumbersome or not valued), yet even the “some information” needs to be valid and credible. All in all, if I had to take a stand on the design structure, it would be to develop as rigorous of an evaluation as possible (i.e., aim toward the medical model of research), while taking into account stakeholder preferences and contextual constraints.

Calibrator D2: Design Purpose.

Design purpose relates to the process and intent of the evaluation. At one extreme is keeping the program or intervention “pure,” that is, not making any changes to the program during the evaluation. At the other extreme is continuous program improvement, such that the program is adjusted on an ongoing basis throughout the evaluation based on formative data. An argument for the former is that if the program is continuously changing, it is difficult to know what the program really is and the extent to which results are “muddied” by strategies that are not consistently implemented. However, an argument on the other extreme is that it is a missed opportunity, and perhaps even unethical, to not make programmatic improvements that would likely improve results for program participants. Important considerations include the following:

In what ways do mid-evaluation program changes affect the interpretability of findings?

In what ways do mid-evaluation program changes affect the replicability of the program?

To what extent should a program make adjustments during an evaluation based on formative data?

In what ways should formative evaluation be used during an evaluation to improve a program?

1122924 – SAGE Publications, Inc. (US) ©

As with all calibrators described in this section, I understand the arguments for and against the extremes. On many calibrators, like design structure, my preference is a range and highly situational. However, with regard to design purpose, I have a strong preference. You do not need to agree with me, and I hope you will develop your own preferences over time with careful consideration and experience. My preference with regard to design purpose is that program evaluation should focus on continuous improvement. I believe it is one of the features of program evaluation that sets it apart from other forms of research. Evaluation is about improving programs and policies, and I think we have the best chance of doing so if we make continual, deliberate programmatic changes based on data, all while carefully documenting those changes.

4.2.2 Role Calibrators

Role calibrators help us to organize our thinking around the function and responsibility of the evaluator. As mentioned in the above section, the context of the evaluation can influence the extent to which an evaluator can implement an evaluation in the preferred manner. However, the two calibrators discussed below can help shape your own ideology around your preferred role as an evaluator.

Calibrator R1: Evaluator Involvement.

Early evaluation viewed the evaluator as a dispassionate, pietistic expert, brought in to pass judgment. While this may seem harsh, evaluators were not seen as partners, collaborators, or friendly visitors. Evaluators were fairly hands-off when it came to the program. Remnants of this view can still be seen in how evaluator visits are perceived by program staff. Evaluators can make program staff nervous, just as we are nervous anytime we feel evaluated. Even though the evaluation is of a program and not an individual, program staff still have a stake in the findings. If the program is not functioning well, program staff may lose responsibility or even their job. However, more recently, evaluators are often partners with program staff. This partnership can take many forms, from the evaluator working only in an advisory capacity to the evaluator working closely with program staff during all phases of the evaluation. Thus, the evaluator involvement calibrator addresses the role of the evaluator and how the evaluator fits within the program. Important considerations include the following:

What role and relationship should an evaluator have with program staff?

To what extent should the evaluator keep firm boundaries between the evaluation and the program? What are the benefits and drawbacks to keeping such boundaries?

To what extent should the evaluator be a full partner with program staff in determining the direction of the program?

A former colleague of mine liked to refer to an evaluator as a critical friend. Evaluators can be a critical friend, a trusted partner, or an external consultant. The difficulty is determining what the most appropriate role is for any given evaluation.

Calibrator R2: Evaluator Responsibility.

An important consideration with regard to evaluator role is the responsibility the evaluator has to a program and to the organization within which that program is implemented. On the one hand, evaluation can be an external activity, with the evaluator’s responsibility to come in, complete the evaluation, provide a report, and leave. On the other end of the spectrum, the evaluator can view evaluation as a capacity-building activity. In such cases, the evaluator seeks to build processes and facilitate data-driven practices that are still in place when the formal evaluation is complete. Important considerations include the following:

To what extent should an evaluation be external/peripheral to the program?

To what extent should evaluators collect the data necessary for the evaluation without interfering with program processes?

How much should an evaluation try to change program processes to incorporate data collection as an ongoing process?

In what ways can and to what extent should evaluators build structures into programs to facilitate a reliance upon data by program staff post-evaluation?

As with design purpose, evaluator responsibility is a calibrator about which I have strong opinions. You may disagree with my opinion and I encourage you to form your own opinion. However, I will share that I believe evaluators have a responsibility to make a difference, not just in the report or recommendations that they leave behind, but in the systems they create during the evaluation. If, as evaluators, we are truly committed to promoting the use of data to make programmatic decisions, we will work to build capacity within programs such that staff are not dependent upon an external evaluator for data-based decision making. We should facilitate an environment of continuous improvement so that data use remains even as the evaluator moves on.

4.2.3 Methods Calibrators

Methods calibrators help us to organize our thinking around the types of data collection methods we value and how we use the methods to orient our evaluation. The two calibrators discussed below can help shape your own ideology around your views on methods and evaluation focus.

Calibrator M1: Data Collection Methods.

A common debate in evaluation, and in all research, is the value of quantitative and qualitative methods. Many evaluators have a strong preference for one over the other, though I daresay most evaluators recognize the usefulness of employing mixed methods, that is, including both quantitative and qualitative methods in an evaluation. Quantitative methods typically allow evaluators to capture information more quickly from more individuals, and large volumes of quantitative data can be analyzed much more quickly than qualitative data. Quantitative measures are also more reliable, whereas it is much more time-consuming for qualitative researchers to ensure adequate reliability in qualitative data. However, qualitative measures can have adequate reliability with structured and consistent data analysis procedures. An example of quantitative analysis is how the multiple-choice items on your SATs can be scored quickly by a machine, and regardless of how many times they are scored, the results would be the same. In addition, the SAT multiple-choice data from thousands of people can be scored at the same time. On the other hand, analysis of the SAT writing responses is more time-consuming and multiple raters are used to score each essay. Yes, any individual scorer can only score one essay at a time. In addition, these raters must go through extensive training to ensure they are consistent in their scoring. The analysis of qualitative data requires similar techniques that might be used to score an essay and such techniques include inter-rater reliability considerations that are not present in quantitative analysis. However, while quantitative methods allow you to analyze more data, more quickly, and with more reliability, they do not provide the kind of rich detail and description that is inherent in qualitative data. Thus, qualitative data can be used to illuminate quantitative findings, such that we can better understand the meaning behind responses and calculations. Quantitative data can help us to determine the generalizability of findings from qualitative data, by enabling us to create closed-ended items on a topic that can be asked of a larger group of people. Thus, important considerations include the following:

1122924 – SAGE Publications, Inc. (US) ©

To what extent do quantitative methods restrict the ability of evaluators to understand a program’s operation and impact at a deeper level?

To what extent do the smaller samples involved in studies using qualitative methods portray a skewed view of a program? How can samples in qualitative studies be constructed so that findings are representative of a larger stakeholder group?

In what ways can both quantitative and qualitative methods be used to provide both depth and breadth to an evaluation?

Calibrator M2: Methods Focus.

With regard to evaluation methods, another consideration for evaluators is how to focus their evaluation. At the heart of methods focus is whether evaluation designs should be constrained by program goals. Ralph Tyler, first introduced in Chapter 2 as the “Father of Evaluation,” laid the groundwork for what most consider to be the first program evaluation approach: objectives-oriented evaluation. While there were other methods that people used to make decisions prior to focusing on objectives, such as expertise-oriented evaluation based on expert opinion and consumer-oriented evaluation intended to make evaluative judgments for public good, objectives oriented was the first evaluation approach geared toward making a value judgment for a specific stakeholder group (Fitzpatrick, Sanders, & Worthen, 2011). Objectives-oriented evaluation focuses on the goals and objectives of a program. Methods are chosen to measure data based on these objectives, and findings are analyzed to determine the extent to which those objectives were met. On the other end of the methods-focus spectrum is Michael Scriven’s goal-free evaluation. Unlike objectives-oriented evaluation, goal-free evaluation is not designed around the specified goals and objectives of a program. Instead, the stated goals and objectives of a program are viewed as incomplete, potentially biased, and a barrier to fully evaluating the program (Scriven, 1991, 2013). Scriven recommends that evaluators examine the program in its entirety, such that both intended and unintended outcomes are measured. So, while there is some element of measuring objectives in a goal-free evaluation, it is not the sole focus to the extent that additional important outcomes are overlooked. For instance, focusing solely on measuring achievement changes based upon a program to increase the rigor of courses might overlook an increase in the number of students who drop out due to frustration. On the other hand, measuring only a program’s goal of increasing youth participation in summer programs might miss the decrease in neighborhood crime by youth during the summer months. Important considerations regarding the methods-focus calibrator include the following:

Objectives-oriented evaluation: an approach to evaluation where the focus of the evaluation is on how well the program met a set of predetermined objectives.

Goal-free evaluation: an approach to evaluation where the evaluation is not constrained by program goals, but rather focuses on the measurement of outcomes, whether intended or unintended; developed by Michael Scriven.

To what extent are a program’s goals worded as strategies the program intends to implement versus the outcomes that would result if those strategies were implemented as planned? What is an evaluator’s responsibility to work with program staff to truly understand a program’s goals, beyond those stated by the program?

How likely is it that the program will have unintended consequences, either positive or negative?

What is an evaluator’s responsibility with regard to evaluation focus? To what extent is an evaluator only obligated to evaluate the objectives of a program as stated by program leadership? To what extent does an evaluator have a responsibility to study other potential impacts of a program beyond the intended goals?

In the Real World … Revisiting the Cambridge-Somerville Youth Study (CSYS): Ideology. The CSYS was introduced in Chapter 2. The purpose of CSYS (Cabot, 1940) was both to prevent juvenile delinquency among boys as well as to study the effectiveness of juvenile delinquency interventions. It is revisited here to illustrate how ideology relates to design, role, and methods.

While it is impossible to truly know what Cabot’s ideology was with regard to evaluation, we can surmise from descriptions of the study where his beliefs might fall along each calibrator continuum. With regard to the design calibrators, the design indicates he favored the medical model of research design and it does not appear that findings were used for program improvement. With regard to role calibrators, the evaluators were external and do not appear to have had much involvement with the program beyond data collection. There is no evidence that CSYS was a capacity-building evaluation. With regard to methods calibrators, while some qualitative methods may have been used, the predominant methods appear to have been quantitative and focused on the objectives of the CSYS.

What if the CSYS evaluators had had a different ideology? Consider each scenario and identify ways that the change may have affected the findings from the study.

SCENARIO 1: Suppose the evaluators chose to forgo a control group and included all youth in the program.

SCENARIO 2: Suppose findings were used throughout the program to improve services for the children involved.

SCENARIO 3: Suppose the evaluator was someone internal to the program.

SCENARIO 4: Suppose the evaluators worked closely with stakeholders throughout the program, building processes for them to collect and analyze their own data.

SCENARIO 5: Suppose the youth were observed and data were collected from these observations, instead of from instruments designed to measure behavior.

1122924 – SAGE Publications, Inc. (US) ©

SCENARIO 6: Suppose the evaluators did not use the stated objectives of the program to drive the study, but instead examined any potential outcome of the program.

Ideally, the relationship between evaluators and stakeholders would be a partnership, such that decisions regarding the focus of an evaluation can be jointly determined. See “In The Real World” for a discussion of how ideology may have influenced the Cambridge-Somerville Youth Study.

4.2.4 Calibrators and Ideology

The six calibrators discussed above are provided for you as areas to reflect upon as you develop your own ideology around evaluation. Figure 4.2 includes a graphical representation of how these calibrators shape ideology—and how ideology, in turn, guides our choices regarding evaluation designs and approaches. On the right side of the diagram are additional influencers that affect our use of evaluation designs and approaches. For instance, resources and context can constrain the types of research designs that might be employed. Evaluator skills and experiences, as well as the degree of access we have to stakeholders, influence the approaches that we are able to take with regard to a particular evaluation. The following two sections will address evaluation design and approaches. The section on designs focuses on how they were shaped by early evaluators and only includes a brief explanation of their purpose (Chapter 8 provides detailed information on evaluation design). Evaluation approaches describe some common evaluation approaches in the field and who contributed to the development of each approach.

Description

Figure 4.2 Evaluation Ideology Calibrators and Influencers

Quick Check 1. How does an evaluator’s ideology affect their choice of evaluation designs and approaches? 2. Do you think an evaluator should build evaluation capacity among the staff of the program they are evaluating? Why or why not? 3. If you had limited resources for an evaluation, would you use qualitative or quantitative methods? Explain your reasoning. 4. Explain the six calibrators that affect an evaluator’s ideology. What are your thoughts on each calibrator? Do you have strong preferences regarding

any of the calibrators?

4.3 EVALUATION DESIGN Ideology influences the evaluation designs we choose to use. In particular, our philosophy regarding the design and methods calibrators drives the overall structure of our evaluation. While there are additional factors that affect and constrain choices regarding evaluation design and methods, our underlying ideology shapes the extent to which we view different research designs as strong or weak. In this section, major contributors to evaluation design will be discussed, within the framework of the evaluation designs themselves. Chapter 8 will explore evaluation design in more detail.

4.3.1 Experimental Designs

Donald Campbell.

Donald Campbell was one of the most critical pioneers in the call for social experimentation in the field of evaluation (Rossi, Lipsey, & Henry, 2018). His groundbreaking work pioneered the application of the experimental model used in psychological research to the evaluation field (Christie & Alkin, 2013). The

1122924 – SAGE Publications, Inc. (US) ©

experimental model of research includes random assignment of subjects to a program/intervention or to a control condition. Campbell’s perspective was that decisions about policy and programs should be made on the basis of experimental research. His perspectives are detailed in his 1969 article “Reforms as Experiments.” For over half a century, his work has guided social science researchers and evaluators on how to conduct rigorous research aimed at establishing causal inference. Causal inference is the ability of evaluators to claim that the program they are evaluating is responsible for the outcomes they measured. One of the most influential books in the field was written by Campbell and his coauthor Julian Stanley. Campbell and Stanley’s (1963) seminal work Experimental and Quasi-Experimental Designs for Research is one that every researcher and evaluator should have in their library. They detail design considerations for randomized controlled experiments, quasi-experiments, and nonexperimental studies. Their work has had a lasting impact on the field of evaluation and has facilitated the use of both experimental and quasi-experimental designs (Shadish & Luellen, 2013). It began a shift in the field, which led to randomized experiments being considered the “gold standard” design for establishing causal inference (Christie & Alkin, 2013).

Causal inference: the ability of an evaluator to claim that the program they are evaluating is responsible for the outcomes they measured; causality can be claimed with experimental designs.

Robert Boruch.

Similar to Campbell’s legacy, Robert Boruch has been instrumental in furthering the use of randomized experiments in the evaluation field. One of Boruch’s (1997) most influential works, Randomized Experiments for Planning and Evaluation, provides a practical guide to randomized experiments for evaluators. Boruch is a strong proponent of using randomized experiments, promotes them as the most effective method for evaluating a program’s effects, and argues that any program can employ randomized experiments to determine their effectiveness. As stated by Christie and Alkin (2013),

1122924 – SAGE Publications, Inc. (US) ©

evaluation design will still yield useful information. Yet a strong logic model within a rigorous evaluation design will enable much stronger conclusions regarding program effectiveness and impact. As you have likely surmised, a weak logic model within a strong evaluation design provides little useful information, just as an unreadable treasure map within a sturdy home brings you no closer to the treasure. That said, in this section you will add strength and depth to your logic model by continuing to build upon the evaluation matrix you began in Chapter 7. Methods and tools will be identified or developed for each indicator on your logic model, addressing the question, How will you collect your data?

Although there are many evaluation methods, most are classified as qualitative, quantitative, or both. Qualitative methods rely primarily on noncategorical, free response, observational, or narrative descriptions of a program, collected through methods such as open-ended survey items, interviews, or observations. Quantitative methods, on the other hand, rely primarily on discrete categories, such as counts, numbers, and multiple-choice responses. Qualitative and quantitative methods reinforce each other in an evaluation, as qualitative data can help to describe, illuminate, and provide a depth of understanding to quantitative findings. For this reason, you may want to choose an evaluation design that includes a combination of qualitative and quantitative methods, commonly referred to as mixed methods. Some common evaluation methods are discussed below and include assessments and tests, surveys and questionnaires, interviews and focus groups, observations, existing data, portfolios, and case studies. Rubrics are also included as an evaluation tool that is often used to score, categorize, or code interviews, observations, portfolios, qualitative assessments, and case studies.

Qualitative methods: evaluation methods that rely on noncategorical data and free response, observational, or narrative descriptions.

Quantitative methods: evaluation methods that rely on categorical or numerical data.

Mixed methods: evaluation methods that rely on both quantitative and qualitative data.

Before delving in to different methods, it is worth mentioning the ways in which the terms assessment and survey are sometimes used and misused. First, while the term “survey” is sometimes used synonymously with “evaluation,” evaluation does not mean survey. A survey is a tool that can be used in an evaluation and it is perhaps one of the most common tools used in evaluation, but it is just one tool nonetheless.

Another terminology confusion is between “assessment” and “evaluation.” These too are often used interchangeably. However, many in the field of evaluation would argue that assessment has a quantitative connotation, while evaluation can be mixed method.

Similarly, the term “measurement” is often used syn

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.