Anticrime/Prevention Program: Part 4 – Management Plan Assignment Instructions? This assignment requires you to write a management plan for the chosen anticrime/prevention program. Cover s

47

THEORY-DRIVEN EVALUATION

4 Keywords evaluability assessment approach program impact theory service utilization plan organizational plan

CHAPTER OUTLINE

Introduction 47

Evaluability Assessment Approach 49

Describing and Producing Program Theory 50

Program Impact Theory 52

Service Utilization Plan 53

Program Organizational Plan 53 Step 1: Defi ne Boundaries 54

Step 2: Explicate Program Theory 55

Step 3: Defi ne Program Goals and Objectives 55

Step 4: Describe the Program Functions, Components, and Activities 56

Step 5: Final Corroboration of the Description of the Program Theory 56

Analyzing Program Theory 57

Link between Program Theory and Social Needs 58

Evaluating Logic and Plausibility of Program Theory 59

Comparing Research with Practice 60

Summary 61

Discussion Questions 61

References 61

Introduction In this chapter, we focus on the concepts and procedures neces-

sary for a criminal justice program evaluator to examine the concep- tualization of a program, also known as program theory . Program theory may be expressed implicitly or explicitly. Either way, it explains why a program does what it does and provides the ratio- nale that doing so will achieve the expected or desired results. At the outset of evaluating program theory, criminal justice evaluators may

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

48 Chapter 4 THEORY-DRIVEN EVALUATION

be dismayed by the incomplete or nonexistence of program theory. Many criminal justice programs are poorly designed with errors that can be traced back to a program’s conceptualization. Th ese errors are the result of neglecting the objectives and theory in the planning stages, or implementation, discussed in Chapter 3 . Th is is not always the fault of program planners. Sometimes the political climate sur- rounding criminal justice programs does not allow for extensive plan- ning. Unfortunately, when this is not the case, the conceptualization of the program theory gets little attention.

In criminal justice, this may be the case. Many times, programs are borne from familiar or “off -the-shelf” services without a clear understanding of the match between the program and services ( Coryn, Noakes, Westine & Schroter, 2011 ). In other words, the match between the program and services and needs is not clear. For instance, crime programs generally have a mix of education and counseling. Th e unfortunate issue is that the underlying assump- tions—individuals will change their behavior due to education or counseling—are not discussed or made explicit. Evidence has shown that criminal behavior is often resistant to change from education or counseling. Th erefore, using only this as an underlying set of assump- tions may not produce the desired results.

Th e rationale of a program and conceptualization deserves a sub- stantial amount of scrutiny in an evaluation. Th e scrutiny should result in an understanding of the criminal justice program’s goals and objectives. Welsh (2006) argues that program objectives must be clearly specifi ed and, optimally, be measurable. However, if these goals and objectives do not match the needs that the program is designed to improve, at best, the program can only be expected to be marginally eff ective ( Welsh & Harris, 2004 ).

A necessary function in assessing the program theory is to artic- ulate it—that is, make sure that the descriptions of concepts (i.e., abstractions from reality), assumptions (i.e., hypotheses), and expec- tations (i.e., ideas for outcomes) are in congruence with the manner in which the program is structured and operated. It is a rare case that a criminal justice program will reveal its theory to an evaluator. While implicit, this occurs because the entire statement of the program the- ory is rarely written down. For instance, a program designed to pre- vent juveniles from joining gangs called Gang Resistance Education and Training (GREAT) has a nine-lesson curriculum that is delivered to middle-school children. Esbensen & Osgood (1999) argue that the curriculum was not developed with any specifi c theory in mind, but they state that curriculum does have its basic etiology from crimino- logical theory. Even when it is written, the explication of the program theory comes from an abstract grant or funding proposal that has not been consulted during implementation or program practice.

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

Chapter 4 THEORY-DRIVEN EVALUATION 49

Criminal justice program evaluators have an important task to understand and clearly delineate the program theory in a manner that can be used for an evaluation. Th is chapter will use two views as guides to help criminal justice program evaluators understand and clearly delineate program theory. First is presenting program theory in a way that represents stakeholders’ views and understanding of the program that makes it workable for a program evaluation through the evaluability assessment approach. Second is describing a program theory for diff erent parts of a program.

Evaluability Assessment Approach Th e fi rst approach is a presentation of an evaluability assess-

ment. An evaluability assessment refers to a method of identifying and describing the program theory by outlining its components and deciphering which of them is measurable ( Welsh, 2006 ). Van Voorhis, Cullen & Applegate (1995) argue that evaluability assessments are designed to ascertain if a program has a sound theoretical basis, has a well-designed treatment protocol, has been implemented as designed, and is suitable for further inquiry such as an outcome evaluation.

Th is type of assessment is important for several reasons. First, the program theory may or may not be clear. Second, the program staff may or may not have diff ering views about the program theory. Understanding the program theory and the views of the program’s staff are important for the evaluator. Knowing this information, the evaluator is able to sort out with the stakeholder the program theory that can be used to evaluate the program.

Performing an evaluability assessment can take many forms. Van Voorhis & Brown (1996) argue that an evaluability assessment requires four steps: 1. Identify the purpose and scope of the assessment. 2. Develop a program template that describes the goals and objec-

tives of the program, the theory underlying the program, and the intended treatment protocol.

3. Validate the program design through interviews and focus groups with program staff members and stakeholders and through obser- vations of program activities.

4. Prepare a report that details the assessment fi ndings and provides the proper recommendations for future evaluation or program improvements. Welsh (2006) takes a diff erent focus in outlining the process for an

evaluability assessment. He argues that the evaluator should do the following: 1. Review the documentation that describes the program. 2. Interview administrators and line staff members.

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

50 Chapter 4 THEORY-DRIVEN EVALUATION

3. Read case fi les. 4. Interview program staff .

Th e evaluability assessment may provide information to improve the program. Smith (1989) argues that evaluability assessments gen- erally show the problems within a program, including: ● Th e need to clarify the target population. ● Th e need to reconceptualize the intervention. ● Th e realization that few objectives are agreed upon by stakehold-

ers and program staff . Th e evaluability assessment can make it clear that more work

is necessary to fi ne tune the program. Welsh (2006) argues that this type of assessment results in a program model that can be reviewed by program administrators and staff for accuracy. Regardless, if the intent is to arrive at information from Smith’s (1989) perspective or Welsh’s (2006) perspective, the evaluability assessment may help pro- gram staff deal with diff erences.

While this type of evaluation may be used for a single program, Matthews, Hubbard & Latessa (2001) see evaluability assessment as a means of improving correctional programming on a large scale. Using the Correctional Program Assessment Inventory (CAPI), Matthews et  al. (2001) examine whether 86 treatment programs are meeting the principles of eff ective intervention. Th e results of this assessment show that 34.1% of the programs studied were unsatisfac- torily meeting the principles of eff ective intervention. Th e main prob- lem with these programs is that they lack integrity. In other words, the evaluability assessment shows that the programs are not being implemented properly. While this perspective is good for improve- ments, it does not necessarily focus on a description of the program theory. Th is perspective, rather, uses the program theory as part of the evaluability assessment to determine the problems with the program. In other words, the program theory may be lost using this model. An evaluability assessment may be fruitful for eliciting pro- gram theory on a program-by-program basis.

Describing and Producing Program Theory Th e second approach is describing and producing program the-

ory because not all evaluations involve the entire program theory. Each part of the program may operate using diff erent theories. Th is approach focuses on describing program theory for diff erent parts of the program rather than the entire program.

Th e evaluation literature has long recognized the importance of program theory and its potential uses for formulating and prioritiz- ing questions, research designs, and interpreting evaluation fi ndings

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

Chapter 4 THEORY-DRIVEN EVALUATION 51

( Chen, 2005; Chen & Rossi, 1980, 1983 ). Chen (1990) writes that pro- gram theory is important because not using this type of theory results in an evaluation that is mechanical or uniform without concern for the theoretical implications of the program’s content, setting, partici- pants, or implementing organizations. In other words, without using program theory, the evaluation is only able to determine if the pro- gram met narrowly defi ned goals rather than satisfi ed theoretical implications. Chen (1990) notes:

Th e question is of whether or not goals are appropriate for the eff ectiveness of the program, or whether these rational goals and procedures could lead to unintended consequences, might not be considered. Evaluators may sometimes be serving only bureaucratic interests and neglect the broader implications for human needs and purposes from the perspective of the stakeholders. To avoid such problems, a new conceptual framework for evaluation should be concerned with value rationality and should provide more insight into the real purposes of a program and its implication for wider social interests. (p. 34)

In criminal justice, evaluations do happen in this manner. For instance, the early evaluations of boot camps as shock incarceration fall into this category. Four- to six-month boot camp–like situations that emphasize military-style discipline and physical exercise served as a correctional alternative to incarceration in the 1990s. Th e early evaluations of this type of program only considered the goals (i.e., reduce recidivism) ( MacKenzie & Shaw, 1993 ) and not the broader theoretical implications (i.e., get tough on crime or self-regulation through self-esteem improvement).

With this in mind, we take the perspective that a program involves a series of interactions between the program staff and target popu- lation. Th ese interactions may involve a multitude of things (e.g., counseling sessions, education sessions, nutrition, medical services, etc.). Th ere are multiple interactions in a program. For instance, the program serves as the organization (i.e., infrastructure, personnel, resources, or activities), and the target population brings their lives (i.e., circumstances) to the organization. Th ese factors may infl uence the success of the program.

Th is perspective brings three components together: impact theory, service utilization plan, and the program’s organizational plan ( Rossi, Lipsey & Freeman, 2004 ). Th e impact theory provides instances about the change process and the change in the conditions. Central to the impact theory is the program’s staff –target population interactions because this is the only way that the change can occur. Th e impact theory may be simple, complex, formal, or informal, but the end result is that change occurred. If change did not occur, then something is faulty about the impact theory.

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

52 Chapter 4 THEORY-DRIVEN EVALUATION

Th e impact of a program does not occur by itself. Programs need a stimulus to provide services to the target population. To provide the services as suffi ciently as possible, program staff turn to the service utilization plan. Th e service utilization plan is vital to a theory-driven evaluation because it contains the assumptions and expectations for reaching the target population. In other words, the service utilization plan provides the proper manner to initiate and terminate the service.

Th e program has to be developed and organized in a manner to provide the intended services. Th e organizational plan provides a sche- matic as to how the program resources, personnel, administration, and general organization are composed and utilized. Th e organizational plan is a series of hypotheses that tie together the infrastructure, per- sonnel, resources, etc. to provide the services as intended. Th e combi- nation of resources and organization come together so that the service may be provided and maintained. Th e organization and services are under the control of program staff . Th ese two things—organization and service—represent the program’s process.

Program Impact Theory Program impact theory is about causality. Th is type of theory

describes the cause-and-eff ect sequence of events that come from the program. Specifi cally, there is a stimulus and certain outcomes that come from the stimulus. Generally, evaluators discuss program impact theory using a causal diagram that outlines the cause-and- eff ect events ( Chen, 1990 ). However, it is relevant for an evaluator to keep in mind that programs rarely have direct control over the social conditions that they are expected to improve, and this means that they will have some indirect eff ect for a benefi t to occur.

Th e most basic form of program impact theory takes place in two stages. In the fi rst stage the services are administered that infl uence some intermediate condition, and in the second stage some improve- ment occurs ( Chen, 2011 ). For example, a program to improve alco- hol abuse will use motivation or attitudinal services to infl uence the intermediate conditions that lead to alcohol abuse. While this type of theoretical premise is worthwhile, many programs operate in a more complex manner. Th e complexity comes with many more stages between the program and the benefi ts, and this may or may not mean more than one path to the benefi t.

Th e key to representing any program impact theory is that each part of the program will have a cause-and-eff ect feature. Th at is, each service will cause a linkage between some other services within the program. In the boot camp example, the impact theory is that disci- pline, exercise, and education will result in less recidivism because the attendees have better levels of self-regulation.

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

Chapter 4 THEORY-DRIVEN EVALUATION 53

Service Utilization Plan Th e service utilization plan provides a clear understanding of how

and why program participants became or will become involved in the program. Th is plan continues to show how the participant will follow through to the point of completing the program. In other words, the service utilization plan provides an outline of program–targets inter- actions from the perspective of the participant, and his or her journey through the program. For instance, a drug treatment program will have a specifi c plan as to how counselors and attendees will interact. Th e plan may consist of days, times, locations, and types of materials.

Program Organizational Plan As the service utilization plan is written from the perspective of

the participant, the program organizational plan is written from the perspective of program management. Th is plan brings together the functions and activities of the program. Further, the program orga- nizational plan provides management with the expectations of the program and the resources (i.e., human, fi nancial, and infrastructure) that are required for the program to function. Key to this plan is the program services that the management or staff are to perform so that the participants are able to reap the intended benefi ts. Th e program organizational plan has to provide information about the sustainabil- ity of the program, including fundraising, personnel management, facilities acquisition and maintenance, and political climate.

A program’s organizational plan may be shown in several diff er- ent ways. Focusing on the interactions between the program and its participants allows the fi rst element—a description of the program’s objectives for the services provided—of the program organizational plan to permeate to the top. If this element does not permeate to the top, a number of questions may assist in this process: ● What are the services? ● How much is to be provided? ● To whom are the services to be provided? ● On what schedule are the services to be provided?

Th e second element of the program organizational plan is to describe the resources and functions that are necessary for the program. Th is includes a description of the personnel with proper credentials and skills, logistics, proper facilities, funding, supervision, or clerical support.

Having been exposed to the issues noted here does not guarantee that it is clear on how to extract program theory. Th e production of program theory is important to develop a high-quality program eval- uation. Program theory may be articulated or spelled out clearly so that everyone understands the theory behind the program. Th is tends

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

54 Chapter 4 THEORY-DRIVEN EVALUATION

to occur when a program comes from a criminological theory. For instance, social learning theory may be a theoretical premise used to reduce instances of gang involvement.

Unfortunately, all program theory is not well articulated. Th is is known as implicit program theory ( Chen, 1990 ). Th is type of theory means the underlying assumptions about program services and prac- tices are not well discussed and articulated, in the context of theory. Th is type of program theory provides evaluators the most trouble.

Implicit program theory requires the evaluator to extract and describe the theory before it can be analyzed or evaluated. Th e eval- uator has to determine the intentions of the program framers and stakeholders about what the program should be doing. Th e extraction of this type of theory works best using a set of steps.

Step 1: Define Boundaries Th e fi rst step of extracting an implicit program theory is defi ning

the boundaries of the program, which are contingent on the scope of the program and the scope of the program staff ’s concerns. An evaluator may identify the boundaries of the program if he or she works from the perspective of the decision makers. Th e decision mak- ers should have some idea of the activities of the program and orga- nizational structures . In other words, the decision makers, especially those who are charged with acting on the results of the evaluation, should have some idea where the other decision makers stand on the reasons why the program exists and how the program should operate.

Th e defi nition of program boundaries has to include all the important activities, events, or resources that have a link with one or more outcomes that are central to the program. Th e evaluator may be able to uncover these boundaries by beginning with the benefi t from the program and working backward to identify the relevant activi- ties and resources that may make a contribution to organizational or programmatic objectives. From this perspective, a drug treatment program at the local or state level could be distilled as a set of activi- ties that are organized by a rehabilitation group to alleviate drug abuse in a specifi ed participant group.

While these two approaches are good starting points, their ratio- nal presentation may oversimplify the process of extracting program theory. Evaluators have to keep in mind that programs may be com- plex. Further, each of the objectives of a program may be diffi cult to establish. Th is puts the evaluator into a position where he or she has to negotiate the defi nition of program boundaries with program staff , stakeholders, and decision makers. Further, evaluators have to recog- nize that the defi nition of boundaries is a fl uid process that requires them to be fl exible.

For example, Esbensen & Osgood (1999) describe the boundaries of the GREAT program. Th e program is designed to reduce instances

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

rv ed

.

Chapter 4 THEORY-DRIVEN EVALUATION 55

of juveniles joining gangs. To address this specifi c need, the GREAT program is bound to middle-school students, and the curriculum is delivered by law enforcement. Here, the boundaries are the age group (i.e., 11–13 year olds) of the students.

Step 2: Explicate Program Theory Not all program theory is original. Some program theory may

come from prior experience of program planners, research, or prac- tice. Th is is a welcomed situation for an evaluator because he or she is able to develop a well-articulated theory. Th e evaluator encoun- ters issues when dealing with an existing program. He or she has to work through the structure and operations of the program to derive the theory. Th is means that the evaluator has to work with stakehold- ers to bring out the theory that comes from their actions and assump- tions. Th e evaluator accomplishes this by producing multiple drafts describing the program theory. He or she presents these drafts to stakeholders and decision makers to ensure that their approxima- tions of the program theory are correct.

Esbensen & Osgood (1999) describe the development of the GREAT program in this manner. In 1991, when the Phoenix, AZ, Police Department introduced GREAT as a school-based program, they used D.A.R.E. (Drug Abuse Resistance Education) as a model to create a nine-week curriculum. D.A.R.E. is a program specifi – cally designed to provide resistance training and education for drug use. Th is program generally consists of a 17-lesson curriculum that is off ered once a week in schools. Both curriculums for GREAT and D.A.R.E. are off ered by trained law enforcement offi cers.

To begin the process of drafting a program theory like this one, the evaluator needs to turn to several sources for information. Th e evalu- ator has to analyze the program documents. Th is can be all reports, mission statements, or goal statements. Th en, the evaluator should interview the stakeholders and decision makers. Th is will provide information from both of these respective sides of the program. Next, the evaluator should perform site visits to observe the functions and circumstances of which the program operates in its natural environ- ment. Finally, the evaluator should consult the criminological litera- ture. Information from these places will assist the evaluator in drafting a reasonable program theory, but the fi nal verdict comes from the review of the program theory by the stakeholders and decision makers.

Step 3: Define Program Goals and Objectives Program goals and objectives are central to program theory. Th ey

provide an understanding of what the program should be accom- plishing. Th e issue that often arises with program goals and objec- tives is that they do not also mesh with mission statements, or they

Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 08:48:26.

C op

yr ig

ht ©

2 01

4. T

ay lo

r &

F ra

nc is

G ro

up . A

ll rig

ht s

re se

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.