Social Science

12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234 12345678901234567890123456789012123456789012345678901234567890121234

Dr. Susan Barkman Purdue University AGAD Room 229 615 W. State Street W. Lafayette, IN. 47907-2053 765/494-8436 [email protected]

Utilizing the

LOGIC MODEL for Program Design

and Evaluation

ID 270

2

Glossary of Important Outcome Measurement Terms

baseline – data about the condition or performance prior to the educational intervention

(i.e. statewide % of low-birth weight; number of vandalism reports; standardized test scores; school dropout rate; number of accidents involving teen drivers; recidivism rate; water quality)

benchmarks – performance data used either as a baseline against which to compare future performance or as a marker to assess progress towards a goal

(i.e., periodic behavior checklist ratings; amount of debt; academic grades)

evaluation – the systematic collection and analysis of information to determine the worth of a curriculum, program or activity

input – resources that go into a program to achieve its goals, outputs and outcomes

(i.e., staff, volunteers, time, money, materials, equipment, technology, partners)

impact – the long-term outcomes of a program

(i.e., social, economic, civic or environmental impact)

outcomes – end results or benefits the participants get from a program. Outcomes can be:

• intended or unintended • positive or negative

Outcomes fall along a continuum from immediate to short-term to long-term. Long- term outcomes are called impact. Examples are listed below.

immediate outcomes – awareness created, knowledge gained, attitudes changed, skills developed, aspirations sparked

short-term outcomes – behavior changed, practice adopted, decisions made, policies changed or adopted, social action initiated

long-term outcomes – social, economic, civic, environmental impact

outcome indicators – the observable, measurable characteristics or changes that represent achievement of an outcome

(i.e., absenteeism, score on an attitude survey or completed homework assignment could all be indicators for developing a positive attitude towards school)

outcome target – numerical objective for a program's level of achievement (i.e., percent, number, quality rating)

outputs – activities, products, services, and participation that result from the inputs

(i.e., activities such as workshops, camps, curriculum, publications, media event, web site, projects, test plots, field days, and the number, characteristics, and reactions of participants)

qualitative methodology – methods that examine, describe or interpret phenomena

(i.e., observation, open-ended interviewing, and document review)

quantitative methodology – refers to an approach involving the use of numerical measurement and data analysis methodology

(i.e., standardized, structured data collection including surveys, closed-ended interviews, tests and behavior checklists)

reliability – the extent to which measures from an instrument are consistent. Reliability is usually measured by a Cronbach Alpha. The closer an alpha is to 1.0, the more reliable the instrument

( i.e., an instrument with a .85 is more reliable than one with .65. No instrument can have a reliability of 1.00)

validity – the extent to which a measure from an instrument actually captures the concept being measured

3

Evaluation – What Is It? Evaluation means different things to different people and takes place in different contexts. Although many definitions have been developed, the one most appropriate for use within the Extension context is: "The systematic collection and analysis of information to determine the worth of a curriculum, program or activity." (Alkin 1990) The word systematic means the evaluation must be planned as an integral part of the design or planning process. It is not just an "event" that occurs at the end. Worth can be a measure of: 1. the quality or satisfaction level leading to

program improvement, or 2. the effectiveness as a means to

demonstrate the impact of the program or that the targeted outcomes have been achieved.

Both kinds of "worth" continue to be important for Extension professionals. However because of the increased accountability demands, this manual will focus primarily on measuring the impact of programs, otherwise called "outcome evaluation."

Why Measure Outcomes? Basically, the answer to that question is to see if programs really make a difference in the lives of the participants and communities. As you gather outcome data, it can be used in a variety of ways:

• To help improve programs and services.

• To be accountable in order to retain or increase funding.

• To gain valuable information to use in decision making and long range planning.

• To focus programs that can really make a difference for participants.

• To determine cost-effectiveness.

• To gain support from the community to aid in sustainability.

• To gain public and professional recognition as a quality program.

• To attract new participants • To recruit and retain talented staff. • To recruit and motivate volunteers. • To take pride in accomplishment.

Remember an outcome measurement provides an ongoing means for educators and specialists to track the percent of participants that achieved the targeted outcomes, but it does not prove that the program, and the program alone, caused the outcomes. This is a key difference between outcome measurement and program impact research, which uses sophisticated statistical methods and random assignment of participants to either the program or to a control group.

Levels of Outcomes Each program is unique and aimed at achieving a range of different outcomes. As you are developing your evaluation plan, it is important to define what you want to evaluate. In some cases, you may only be interested in finding out how participants responded to your teaching style or how satisfied they were with a program. This is good information to help you improve the program next time. In other cases, you want to document the impact the program had on the participants or whether the program achieved the targeted outcomes. Don't expect to measure impact from a single workshop or behavior changes from a program with limited contact hours (i.e., 45-minute school enrichment program, 1 1/2-hour training). The point is to tailor your evaluation to fit the program. Remember, not all Extension work needs to be formally evaluated. Be selective and think about what information is needed and how it will be used. If you are interested in outcome evaluation, here's a framework for thinking about different levels of outcomes.

4

Outcomes can be classified under three major levels: learning, action and impact. The specific outcomes you can measure are as follows. LEARNING LEVEL Immediate Awareness Created Knowledge Gained Attitudes Changed Skills Developed Aspirations Sparked ACTION LEVEL Behavior Changed Practice Adopted Decisions Made Policies Changed or Adopted Social Action Initiated IMPACT LEVEL Social Impact Economic Impact Civic Impact Environmental Impact Long-Term It is important to determine the level of outcome you desire to reach as you plan your program. Your evaluation should be an integral part of the program planning process and not an afterthought. Remember, outcomes may be positive, negative, or neutral; intended or unintended.

Program Logic Model One effective method for linking program design and evaluation is the program logic model. A program logic model is a picture of how your program works – the theory and assumptions underlying the program. This model provides a roadmap for your program, outlining how it is expected to work, what activities need to come before others, and how desired outcomes are achieved. In simple terms, it gives a visual picture of what inputs and outputs are needed to achieve the desired outcomes.

INPUTS What resources and contributions by Extension and others are needed to achieve our goals?

• people (staff, partners, volunteers) • time • money • materials • equipment • technology

These inputs are converted into OUTPUTS What activities do we need to conduct to ensure our goals are met?

• workshops • meetings • publications • media events • web site • test plots • field days • projects

Who needs to participate, be involved or reached?

• characteristics (youth, parents, leaders, mothers on WIC, etc.)

• number • their reactions

These outputs are intended to achieve certain OUTCOMES These are the changes that occur for children, youth, families, groups, communities, organizations, and systems. Outcomes range from immediate to long-range; from learning to action to social, economic, and environmental impact. These are listed in the first column of this page.

Logic Model for Program Design and Evaluation

Assumptions: 1. 2. 3.

Adapted from E. Taylor-Powell, 1999

S. Barkman and K. Machtmes – Purdue University (revised 2002)

INPUTS OUTPUTS OUTCOMES – IMPACT

Activities Participation

ENVIRONMENT Influential Factors

What do we need to achieve our goals?

What do we have to do to ensure our goals are met?

Who needs to – participate? – be involved? – be reached?

What do we think the participants will know, feel, or be able to do after participation in the program?

How do we think the participants will behave or act differently after participation in the program?

What kind of impact can result if the participants behave or act differently after participation in the program?

LEARNING ACTION IMPACT

Staff Volunteers Time Money Materials Equipment Technology Partners

Workshops Meetings Camps Curriculum Publications Media Web Site Projects Field Days

Number Characteristics Reactions

Awareness Knowledge Attitudes Skills Aspirations

Behavior Practice Decisions Policies Social Action

Social Economic Civic Environmental

PROGRAM DESIGN AND PLANNING

MEASURING PROGRAM IMPACT

IMPACT Social. Economic, Environmental, Civic 7. Bennett’s – SEE (Social, Economic, and Environmental)

ACTION Behavior, Practice, Decisions, Policies, Social Action 6. Bennett’s – Practice (patterns of behavior,

procedures or action)

LEARNING Awareness, Knowledge, Attitudes, Skills, Aspirations 5. Bennett’s – KASA (knowledge, attitude, skills,

aspirations)

Reactions People’s reactions to a program (degree or positive or negative interest in topics addressed, their acceptance of instructors, and their attraction to the educational methods

4. Bennett’s – Reactions

Participants Number and Characteristics of people involved 3. Bennett’s – Participants

Activities W orkshops, Meetings, Camps, Curriculum, Publications, Media, W eb Site, Projects, Test Plots, Field Days

2. Bennett’s – Activities

Resources Staff, Volunteers, Time, Money, Materials, Equipment, Technology, Partners

1. Bennett’s – Activities

OUTCOMES

OUTPUTS

INPUTS

Adapted from Bennett & Rockwell TOP Hierarchy

How Bennett’s TOP Hierarchy Fits Into Logic Model

7

How Does the Logic Model Work? The logic model is really a sequence that shows the logical relationship between inputs, outputs, and outcomes. IF then IF then IF INPUTS OUTPUTS OUTCOMES A logic model displays the chain of events that will effect changes and achieve your targeted outcomes. You might think about this as a series of if-then relationships. Here are some examples of this relationship. If the "Have a Healthy Baby program" teaches about the importance of good nutrition and avoidance of smoking and drugs for a baby's health (outputs), then pregnant teens acquire the knowledge, change their attitude, and gain skills to eat more nutritiously (immediate outcomes). If pregnant teens know about and want to eat nutritiously, then they will eat nutritious meals (short-term outcomes). If they eat more nutritiously, then their baby will be born healthy (longer-term outcomes.) If teens receive SERIES training on giving effective presentations and teaching younger youth (outputs), then they will acquire the knowledge and skills to be a good instructor (immediate outcomes). If teens have the knowledge and are given the opportunity to practice teaching, then they will improve their communication and teaching skills and become effective instructors (short-term outcomes). If the pesticide application training program teaches farmers how to apply the correct amount of pesticides per acre (outputs), then they will gain the knowledge and skills to use pesticides appropriately (immediate outcomes). If farmers apply pesticides appropriately, then pesticide runoff is reduced (short-term outcomes). If pesticide runoff is reduced, then the environment and ground water will not be polluted (longer-term outcomes).

Underlying each of these if-then relationships are a number of assumptions. Assumptions are the beliefs we have about the program and the way we think it will work; principles that are guiding the program. In addition, you need to consider the environment (context) in which this relationship is occurring. IF then IF then IF INPUTS OUTPUTS OUTCOMES Assumptions: 1. 2. Environment 3.

In the first example, there are two major assumptions: 1. that good nutrition and avoidance of

caffeine, alcohol, cigarettes, and drugs will produce a healthy baby.

2. that a teen mother's desire to have a healthy baby is stronger than the desire to eat junk food, drink caffeine or alcohol, smoke, or use drugs.

In the second example, there is one major assumption: 1. that SERIES training provides all the

information and practice needed to be an effective instructor.

In the third example, there two major assumptions: 1. that over application of pesticides on a

field increases the possibility of pesticide contamination of the ground water.

2. that pesticide runoff pollutes the ground water and the environment.

When you are developing your logic model, think about the underlying assumptions.

• Are they realistic and sound? • What evidence do you have to

support your assumptions?

You also need to think about the environment (the context and external conditions) in which the program exists and influences the success of the program.

8

Why Create a Logic Model? There are several reasons why you may want to create a logic model before you begin implementing your program.

• It can be a tool to use with stakeholders to build consensus around what a program's inputs, outputs (activities and participation), and outcomes look like.

• It creates a sense of ownership among stakeholders.

• It helps bring detail to broad, fuzzy goals. • It shows the "chain of events” that link

inputs to results. • It summarizes the key elements of your

program and how it is intended to work. • It clarifies the difference between activities

and outcomes. • It provides an easy way to monitor your

program's progress and make needed changes.

• It helps focus the evaluation. • It can be used to easily explain the

program to others or funders. • It can be used in funding proposals.

Benefits to Using a Logic Model

• Helps identify appropriate evaluation questions.

• Summarizes complex program to communicate with internal and external stakeholders.

• Builds underlying rationale and expectations.

• Identifies gaps in logic and uncertain assumptions.

Cautions

� The logic model represents a reality that is somewhat inaccurate:

– programs are not linear – programs are dynamic interrelationships that rarely follow a sequential order

� Logic model focuses on expected outcomes, making it easy to miss unexpected outcomes.

� There is a challenge of causal attribution – many factors influence outcomes

(Taylor-Powell, 1999)

Deciding What Outcome(s)

to Measure Determining what outcome(s) to measure is not always an easy decision. There is no required number of outcomes for a program. • The more immediate the outcome, the

more influence a program generally has on its achievement. (United Way, 1996)

• Conversely, the farther the outcome

occurs from the program, the less direct influence a program has over its achievement and the greater the likelihood that other, extraneous forces have intervened. (United Way, 1996)

It is important to determine what the goal of interest is for you and your stakeholders by asking the question, "What do you want to have happen as a result of this program, curriculum, or activity?" That then becomes your "targeted outcome(s).”

Checklist for Outcomes

An outcome should meet all three tests. � Is it reasonable to believe the program can

influence the outcome in a non-trivial way? � Would measurement of this outcome help

identify program successes and help point out shortcomings?

� Will the program's various stakeholders

accept this as a valid outcome of the program?

Once you have examined each outcome to see if it meets the above tests, review your logic model using the checklist on the next page. Then you are ready to determine how you will measure your targeted outcomes.

9

Checklist for Your Program Logic Model and Outcomes

Does the Logic Model:

Include a listing of all inputs that will be needed for this program? (staff, volunteers, time, money, materials, equipment, technology, partners) Include details of the activities listed? (duration, days per week, month, etc; number of sessions; contact hours; location,

etc.) Include a list of the characteristics and intended number of targeted

participants? (youth ages 8-12, pregnant teens, families on welfare, etc.) Make sequential and logical connections between inputs (activities,

participation), outputs, and outcomes? ( 'If" and "then" statements are sequential and logical)

Do Targeted Outcome(s): Help fulfill Extension's mission? Represent meaningful benefits or changes for participants? Seem reasonable as a result of program participants in a non-trivial

way? Clearly define the intended scope of the program's influence? Help educators identify both points of success and problems the

program can correct? Provide data that is likely to be effective in communicating benefits to

stakeholders?

If all of the above items are checked, then you are ready to specify the indicators for your outcomes and where and how you will collect the data.

Susan J. Barkman, Purdue University (2000)

10

Specifying the Indicators

For Your Outcomes Now that you have identified the outcomes of your program, it is time to determine how you will know if those outcomes have been achieved. This information is called the outcome indicator. Specifying an outcome indicator requires deciding:

• the observable, measurable characteristics or change that will represent achievement of the outcome.

• the specific statistic(s) the program will calculate to summarize its level of achievement (i.e., number and percent attaining outcome).

(United Way, 1996) Typically, one to three indicators should be identified for each outcome. Some outcomes are straightforward and require only one indicator. For example, an outcome of improvement of academic grades has just one indicator: comparison of two semesters of grades on a report card. Outcome indicators must be observable and measurable. If a condition is not observable and measurable, it may relate to the outcome, but it is not useful as an indicator. Indicators must be unambiguous. Terms such as "

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.