Students will complete the Checklist for Determining When to Conduct an Evaluation? on page 272 of FSW and will write a paragraph providing a rationale in favor of or against conducting

Program Evaluation

Alternative Approaches and Practical Guidelines

FOURTH EDITION

Jody L. Fitzpatrick University of Colorado Denver

James R. Sanders Western Michigan University

Blaine R. Worthen Utah State University

Boston Columbus Indianapolis New York San Francisco Upper Saddle River Amsterdam Cape Town Dubai London Madrid Milan Munich Paris Montreal Toronto

Delhi Mexico City São Paulo Sydney Hong Kong Seoul Singapore Taipei Tokyo

Vice President and Editor in Chief: Jeffery W. Johnston Senior Acquisitions Editor: Meredith D. Fossel Editorial Assistant: Nancy Holstein Vice President, Director of Marketing: Margaret Waples Senior Marketing Manager: Christopher D. Barry Senior Managing Editor: Pamela D. Bennett Senior Project Manager: Linda Hillis Bayma Senior Operations Supervisor: Matthew Ottenweller Senior Art Director: Diane Lorenzo Cover Designer: Jeff Vanik Cover Image: istock Full-Service Project Management: Ashley Schneider, S4Carlisle Publishing Services Composition: S4Carlisle Publishing Services Printer/Binder: Courier/Westford Cover Printer: Lehigh-Phoenix Color/Hagerstown Text Font: Meridien

Credits and acknowledgments borrowed from other sources and reproduced, with permission, in this textbook appear on appropriate page within text.

Every effort has been made to provide accurate and current Internet information in this book. However, the Internet and information posted on it are constantly changing, so it is inevitable that some of the Internet addresses listed in this textbook will change.

Copyright © 2011, 2004, 1997 Pearson Education, Inc., Upper Saddle River, New Jersey 07458. All rights reserved. Manufactured in the United States of America. This publication is protected by Copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission(s) to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, 501 Boylston Street, Suite 900, Boston, MA 02116, fax: (617) 671-2290, email: [email protected]

Library of Congress Cataloging-in-Publication Data

Fitzpatrick, Jody L. Program evaluation: alternative approaches and practical guidelines / Jody L. Fitzpatrick, James R.

Sanders, Blaine R. Worthen. p. cm.

ISBN 978-0-205-57935-8 1. Educational evaluation—United States. 2. Evaluation research (Social action programs)—

United States. 3. Evaluation—Study and teaching—United States. I. Sanders, James R. II. Worthen, Blaine R. III. Worthen, Blaine R. Program evaluation. IV. Title.

LB2822.75.W67 2011 379.1’54—dc22

2010025390 10 9 8 7 6 5 4 3 2

ISBN 10: 0-205-57935-3 ISBN 13: 978-0-205-57935-8

Jody Fitzpatrick has been a faculty member in public administration at the Uni- versity of Colorado Denver since 1985. She teaches courses in research methods and evaluation, conducts evaluations in many schools and human service settings, and writes extensively about the successful practice of evaluation. She has served on the Board of the American Evaluation Association and on the editorial boards of the American Journal of Evaluation and New Directions for Evaluation. She has also served as Chair of the Teaching of Evaluation Topical Interest Group at the American Evaluation Association and has won a university-wide teaching award at her university. In one of her recent publications, Evaluation in Action: Interviews with Expert Evaluators, she uses interviews with expert evaluators on one evaluation to talk about the decisions that evaluators face as they plan and conduct evaluations and the factors that influence their choices. She is currently evaluating the chang- ing roles of counselors in middle schools and high schools and a program to help immigrant middle-school girls to achieve and stay in school. Her international work includes research on evaluation in Spain and Europe and, recently, she has spoken on evaluation issues to policymakers and evaluators in France, Spain, Denmark, Mexico, and Chile.

James Sanders is Professor Emeritus of Educational Studies and the Evaluation Center at Western Michigan University where he has taught, published, consulted, and conducted evaluations since 1975. A graduate of Bucknell University and the University of Colorado, he has served on the Board and as President of the American Evaluation Association (AEA) and has served as Chair of the Steering Committee that created the Evaluation Network, a predecessor to AEA. His publications in- clude books on school, student, and program evaluation. He has worked exten- sively with schools, foundations, and government and nonprofit agencies to develop their evaluation practices. As Chair of the Joint Committee on Standards for Educational Evaluation, he led the development of the second edition of The Program Evaluation Standards. He was also involved in developing the concepts of applied performance testing for student assessments, cluster evaluation for program evaluations by foundations and government agencies, and mainstreaming evaluation for organizational development. His international work in evaluation has been concentrated in Canada, Europe, and Latin America. He received distinguished ser- vice awards from Western Michigan University, where he helped to establish a PhD program in evaluation, and from the Michigan Association for Evaluation.

About the Authors

iii

Blaine Worthen is Psychology Professor Emeritus at Utah State University, where he founded and directed the Evaluation Methodology PhD program and the West- ern Institute for Research and Evaluation, conducting more than 350 evaluations for local and national clients in the United States and Canada. He received his PhD from The Ohio State University. He is a former editor of Evaluation Practice and founding editor of the American Journal of Evaluation. He served on the American Evaluation Association Board of Directors and received AEA’s Myrdal Award for Outstanding Evaluation Practitioner and AERA’s Best Evaluation Study Award. He has taught university evaluation courses (1969–1999), managed federally man- dated evaluations in 17 states (1973–1978), advised numerous government and private agencies, and given more than 150 keynote addresses and evaluation workshops in the United States, England, Australia, Israel, Greece, Ecuador, and other countries. He has written extensively in evaluation, measurement, and as- sessment and is the author of 135 articles and six books. His Phi Delta Kappan arti- cle, “Critical Issues That Will Determine the Future of Alternative Assessment,” was distributed to 500 distinguished invitees at the White House’s Goals 2000 Conference. He is recognized as a national and international leader in the field.

iv About the Authors

The twenty-first century is an exciting time for evaluation. The field is growing. People—schools, organizations, policymakers, the public at large—are interested in learning more about how programs work: how they succeed and how they fail. Given the tumult experienced in the first decade of this century, many peo- ple are interested in accountability from corporations, government, schools, and nonprofit organizations. The fourth edition of our best-selling textbook is designed to help readers consider how evaluation can achieve these purposes. As in previ- ous editions, our book is one of the few to introduce readers to both the different approaches to evaluation and practical methods for conducting it.

New to This Edition

The fourth edition includes many changes:

• A new chapter on the role of politics in evaluation and ethical considerations. • A new and reorganized Part Two that presents and discusses the most current

approaches and theories of evaluation. • An increased focus on mixed methods in design, data collection, and analysis. • Links to interviews with evaluators who conducted an evaluation that illus-

trates the concepts reviewed in that chapter, as they discuss the choices and challenges they faced.

• A discussion of how today’s focus on performance measurement, outcomes, impacts, and standards have influenced evaluation.

• New sections on organizational learning, evaluation capacity building, mainstreaming evaluation, and cultural competence––trends in evaluation and organizations.

Evaluation, today, is changing in a variety of ways. Policymakers, managers, citizens, and consumers want better tracking of activities and outcomes. More importantly, many want a better understanding of social problems and the programs and policies being undertaken to reduce these problems. Evaluation in many forms, including performance measurement and outcome or impact assessments, is ex- panding around the globe. People who work in organizations are also interested in evaluation as a way to enhance organizational learning. They want to know how well they’re doing, how to tackle the tough problems their organizations address, and how to improve their performance and better serve their clients and their

Preface

v

community. Many different methods are being developed and used: mixed meth- ods for design and data collection, increased involvement of new and different stakeholders in the evaluation process, expanded consideration of the potential uses and impacts of evaluation, and more effective and diverse ways to communicate findings. As evaluation expands around the world, the experiences of adapting eval- uation to different settings and different cultures are enriching the field.

In this new edition, we hope to convey to you the dynamism and creativity involved in conducting evaluation. Each of us has many years of experience in conducting evaluations in a variety of settings, including schools, public welfare agencies, mental health organizations, environmental programs, nonprofit organ- izations, and corporations. We also have years of experience teaching students how to use evaluation in their own organizations or communities. Our goal is, and always has been, to present information that readers can use either to conduct or to be a participant in evaluations that make a difference to their workplace, their clients, and their community. Let us tell you a bit more about how we hope to do that in this new edition.

Organization of This Text

The book is organized in four parts. Part One introduces the reader to key concepts in evaluation; its history and current trends; and ethical, political, and interper- sonal factors that permeate and transcend all phases of evaluation. Evaluation dif- fers from research in that it is occurring in the real world with the goal of being used by non-researchers to improve decisions, governance, and society. As a result, evaluators develop relationships with their users and stakeholders and work in a political environment in which evaluation results compete with other demands on decision makers. Evaluators must know how to work in such envi- ronments to get their results used. In addition, ethical challenges often present themselves. We find the ways in which evaluation differs from research to be both challenging and interesting. It is why we chose evaluation as our life’s work. In Part One, we introduce you to these differences and to the ways evaluators work in this public, political context.

In Part Two, we present several different approaches, often called models or theories, to evaluation. (Determining whether objectives or outcomes have been achieved isn’t the only way to approach evaluation!) Approaches influence how evaluators determine what to study and how they involve others in what they study. We have expanded our discussions of theory-based, decision-oriented, and participatory approaches. In doing so, we describe new ways in which evaluators use logic models and program theories to understand the workings of a program. Participatory and transformative approaches to empowering stakeholders and creating different ways of learning are described and contrasted. Evaluators must know methodology, but they also must know about different approaches to eval- uation to consciously and intelligently choose the approach or mix of approaches that is most appropriate for the program, clients, and stakeholders and context of their evaluation.

vi Preface

In Parts Three and Four, the core of the book, we describe how to plan and carry out an evaluation study. Part Three is concerned with the planning stage: learning about the program, conversing with stakeholders to learn purposes and consider future uses of the study, and identifying and finalizing evaluation questions to guide the study. Part Three teaches the reader how to develop an eval- uation plan and a management plan, including timelines and budgets for conduct- ing the study. In Part Four, we discuss the methodological choices and decisions evaluators make: selecting and developing designs; sampling, data collection, and analysis strategies; interpreting results; and communicating results to others. The chapters in each of these sections are sequential, representing the order in which decisions are made or actions are taken in the evaluation study. We make use of extensive graphics, lists, and examples to illustrate practice to the reader.

This Revision

Each chapter has been revised by considering the most current books, articles, and reports. Many new references and contemporary examples have been added. Thus, readers are introduced to current controversies about randomized control groups and appropriate designs for outcome evaluations, current discussions of political influences on evaluation policies and practices, research on participative approaches, discussions of cultural competency and capacity building in organiza- tions, and new models of evaluation use and views on interpreting and dissemi- nating results.

We are unabashedly eclectic in our approach to evaluation. We use many different approaches and methods––whatever is appropriate for the setting––and encourage you to do the same. We don’t advocate one approach, but instruct you in many. You will learn about different approaches or theories in Part Two and different methods of collecting data in Parts Three and Four.

To facilitate learning, we have continued with much the same pedagogical structure that we have used in past editions. Each chapter presents information on current and foundational issues in a practical, accessible manner. Tables and figures are used frequently to summarize or illustrate key points. Each chapter begins with Orienting Questions to introduce the reader to some of the issues that will be covered in the chapter and concludes with a list of the Major Concepts and Theories reviewed in the chapter, Discussion Questions, Application Exercises, and a list of Suggested Readings on the topics discussed.

Rather than using the case study method from previous editions, we thought it was time to introduce readers to some real evaluations. Fortunately, while Blaine Worthen was editor of American Journal of Evaluation, Jody Fitzpatrick wrote a column in which she interviewed evaluators about a single evaluation they had conducted. These interviews are now widely used in teaching about evaluation. We have incorporated them into this new edition by recommending the ones that illustrate the themes introduced in each chapter. Readers and instructors can choose either to purchase the book, Evaluation in Action (Fitzpatrick, Christie, & Mark, 2009), as a case companion to this text or to access many of the interviews

Preface vii

through their original publication in the American Journal of Evaluation. At the end of each chapter, we describe one to three relevant interviews, citing the chapter in the book and the original source in the journal.

We hope this book will inspire you to think in a new way about issues—in a questioning, exploring, evaluative way—and about programs, policy, and organi- zational change. For those readers who are already evaluators, this book will pro- vide you with new perspectives and tools for your practice. For those who are new to evaluation, this book will make you a more informed consumer of or participant in evaluation studies or, perhaps, guide you to undertake your own evaluation.

Acknowledgments

We would like to thank our colleagues in evaluation for continuing to make this such an exciting and dynamic field! Our work in each revision of our text has reminded us of the progress being made in evaluation and the wonderful insights of our colleagues about evaluation theory and practice. We would also like to thank Sophia Le, our research assistant, who has worked tirelessly, creatively, and diligently to bring this manuscript to fruition. We all are grateful to our families for the interest and pride they have shown in our work and the patience and love they have demonstrated as we have taken the time to devote to it.

viii Preface

Contents

PART ONE • Introduction to Evaluation 1

1 Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions 3

Informal versus Formal Evaluation 5

A Brief Definition of Evaluation and Other Key Terms 6

Differences in Evaluation and Research 9

The Purposes of Evaluation 13

Roles and Activities of Professional Evaluators 16

Uses and Objects of Evaluation 18

Some Basic Types of Evaluation 20

Evaluation’s Importance—and Its Limitations 32

2 Origins and Current Trends in Modern Program Evaluation 38

The History and Influence of Evaluation in Society 38

1990–The Present: History and Current Trends 49

3 Political, Interpersonal, and Ethical Issues in Evaluation 64

Evaluation and Its Political Context 65

Maintaining Ethical Standards: Considerations, Issues, and Responsibilities for Evaluators 78

ix

PART TWO • Alternative Approaches to Program Evaluation 109

4 Alternative Views of Evaluation 111

Diverse Conceptions of Program Evaluation 113

Origins of Alternative Views of Evaluation 114

Classifications of Evaluation Theories or Approaches 120

5 First Approaches: Expertise and Consumer-Oriented Approaches 126

The Expertise-Oriented Approach 127

The Consumer-Oriented Evaluation Approach 143

6 Program-Oriented Evaluation Approaches 153

The Objectives-Oriented Evaluation Approach 154

Logic Models and Theory-Based Evaluation Approaches 159

How Program-Oriented Evaluation Approaches Have Been Used 164

Strengths and Limitations of Program-Oriented Evaluation Approaches 166

Goal-Free Evaluation 168

7 Decision-Oriented Evaluation Approaches 172

Developers of Decision-Oriented Evaluation Approaches and Their Contributions 173

The Decision-Oriented Approaches 173

How the Decision-Oriented Evaluation Approaches Have Been Used 184

Strengths and Limitations of Decision-Oriented Evaluation Approaches 184

x Contents

8 Participant-Oriented Evaluation Approaches 189

Evolution of Participatory Approaches 190

Developers of Participant-Oriented Evaluation Approaches and Their Contributions 191

Participatory Evaluation Today: Two Streams and Many Approaches 199

Some Specific Contemporary Approaches 205

How Participant-Oriented Evaluation Approaches Have Been Used 220

Strengths and Limitations of Participant-Oriented Evaluation Approaches 223

9 Other Current Considerations: Cultural Competence and Capacity Building 231

The Role of Culture and Context in Evaluation Practice and Developing Cultural Competence 232

Evaluation’s Roles in Organizations: Evaluation Capacity Building and Mainstreaming Evaluation 235

10 A Comparative Analysis of Approaches 243

A Summary and Comparative Analysis of Evaluation Approaches 243

Cautions About the Alternative Evaluation Approaches 244

Contributions of the Alternative Evaluation Approaches 248

Comparative Analysis of Characteristics of Alternative Evaluation Approaches 249

Eclectic Uses of the Alternative Evaluation Approaches 251

PART THREE • Practical Guidelines for Planning Evaluations 257

11 Clarifying the Evaluation Request and Responsibilities 259

Understanding the Reasons for Initiating the Evaluation 260

Conditions Under Which Evaluation Studies Are Inappropriate 265

Contents xi

Determining When an Evaluation Is Appropriate: Evaluability Assessment 268

Using an Internal or External Evaluator 271

Hiring an Evaluator 277

How Different Evaluation Approaches Clarify the Evaluation Request and Responsibilities 281

12 Setting Boundaries and Analyzing the Evaluation Context 286

Identifying Stakeholders and Intended Audiences for an Evaluation 287

Describing What Is to Be Evaluated: Setting the Boundaries 290

Analyzing the Resources and Capabilities That Can Be Committed to the Evaluation 304

Analyzing the Political Context for the Evaluation 307

Variations Caused by the Evaluation Approach Used 309

Determining Whether to Proceed with the Evaluation 310

13 Identifying and Selecting the Evaluation Questions and Criteria 314

Identifying Useful Sources for Evaluation Questions: The Divergent Phase 315

Selecting the Questions, Criteria, and Issues to Be Addressed: The Convergent Phase 328

Specifying the Evaluation Criteria and Standards 332

Remaining Flexible during the Evaluation: Allowing New Questions, Criteria, and Standards to Emerge 336

14 Planning How to Conduct the Evaluation 340

Developing the Evaluation Plan 342

Specifying How the Evaluation Will Be Conducted: The Management Plan 358

xii Contents

Establishing Evaluation Agreements and Contracts 367

Planning and Conducting the Metaevaluation 368

PART FOUR • Practical Guidelines for Conducting and Using Evaluations 379

15 Collecting Evaluative Information: Design, Sampling, and Cost Choices 381

Using Mixed Methods 383

Designs for Collecting Descriptive and Causal Information 387

Sampling 407

Cost Analysis 411

16 Collecting Evaluative Information: Data Sources and Methods, Analysis, and Interpretation 418

Common Sources and Methods for Collecting Information 419

Planning and Organizing the Collection of Information 443

Analysis of Data and Interpretation of Findings 444

17 Reporting Evaluation Results: Maximizing Use and Understanding 453

Purposes of Evaluation Reporting and Reports 454

Different Ways of Reporting 455

Important Factors in Planning Evaluation Reporting 456

Key Components of a Written Report 469

Suggestions for Effective Oral Reporting 476

A Checklist for Good Evaluation Reports 479

How Evaluation Information Is Used 479

Contents xiii

18 The Future of Evaluation 490

The Future of Evaluation 490

Predictions Concerning the Profession of Evaluation 491

Predictions Concerning the Practice of Evaluation 493

A Vision for Evaluation 496

Conclusion 497

Appendix A The Program Evaluation Standards and Guiding Principles for Evaluators 499

References 505

Author Index 526

Subject Index 530

xiv Contents

I

Introduction to Evaluation

Part

1

This initial section of our text provides the background necessary for the begin- ning student to understand the chapters that follow. In it, we attempt to accom- plish three things: to explore the concept of evaluation and its various meanings, to review the history of program evaluation and its development as a discipline, and to introduce the reader to some of the factors that influence the practice of evaluation. We also acquaint the reader with some of the current controversies and trends in the field.

In Chapter 1, we discuss the basic purposes of evaluation and the varying roles evaluators play. We define evaluation specifically, and we introduce the reader to several different concepts and distinctions that are important to evalua- tion. In Chapter 2, we summarize the origins of today’s evaluation tenets and prac- tices and the historical evolution of evaluation as a growing force in improving our society’s public, nonprofit, and corporate programs. In Chapter 3, we discuss the political, ethical, and interpersonal factors that underlie any evaluation and em- phasize its distinction from research.

Our intent in Part One is to provide the reader with information essential to understanding not only the content of the sections that follow but also the wealth of material that exists in the literature on program evaluation. Although the con- tent in the remainder of this book is intended to apply primarily to the evaluation of programs, most of it also applies to the evaluation of policies, products, and processes used in those areas and, indeed, to any object of an evaluation. In Part Two we will introduce you to different approaches to evaluation to enlarge your understanding of the diversity of choices that evaluators and stakeholders make in undertaking evaluation.

This page intentionally left blank

Evaluation’s Basic Purpose, Uses, and Conceptual Distinctions

Orienting Questions

1. What is evaluation? Why is it important?

2. What is the difference between formal and informal evaluation?

3. What are some purposes of evaluation? What roles can the evaluator play?

4. What are the major differences between formative and summative evaluations?

5. What questions might an evaluator address in a needs assessment, a process evaluation, and an outcome evaluation?

6. What are the advantages and disadvantages of an internal evaluator? An external evaluator?

3

1

The challenges confronting our society in the twenty-first century are enormous. Few of them are really new. In the United States and many other countries, the public and nonprofit sectors are grappling with complex issues: educating children for the new century; reducing functional illiteracy; strengthening families; train- ing people to enter or return to the workforce; training employees who currently work in an organization; combating disease and mental illness; fighting discrimi- nation; and reducing crime, drug abuse, and child and spouse abuse. More recently, pursuing and balancing environmental and economic goals and working to ensure peace and economic growth in developing countries have become prominent concerns. As this book is written, the United States and many countries around

4 Part I • Introduction to Evaluation

the world are facing challenging economic problems that touch every aspect of so- ciety. The policies and programs created to address these problems will require evaluation to determine which solutions to pursue and which programs and poli- cies are working and which are not. Each new decade seems to add to the list of challenges, as society and the problems it confronts become increasingly complex.

As society’s concern over these pervasive and perplexing problems has intensified, so have its efforts to resolve them. Collectively, local, regional, national, and international agencies have initiated many programs aimed at

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.