Behavior Safety is an important component of keeping employees and clients safe. Below is a vignette of a scenario. Please state how you would addre

SAFETY SCIENCE

Safety Science 32 (1999) 1±18

Long-term evaluation of a behavior-based method for improving safety performance:

a meta-analysis of 73 interrupted time-series replications

T.R. Krause*, K.J. Seymour, K.C.M. Sloat Behavioral Science Technology, 417 Bryant Circle, Ojai, CA 93023, USA

Accepted 22 December 1998

Abstract

Research and applications of behavioral principles have established behavior-based safety initiatives as potentially e•ective solutions to certain occupational health and safety chal- lenges. The present study adds to the existing literature a longitudinal evaluation of an employee-driven behavior-based accident prevention initiative implemented in industrial set- tings. Up to 5 years of injury data from 73 companies, drawn from a target population of 229 companies who implemented behavior-based safety, were examined. Comparisons of pre- to post-initiative incident levels across groups revealed a signi®cant decrease in incidents fol- lowing the behavior-based safety implementation. E•ect sizes were estimated from the aver- age percentage reduction from baseline. The average reduction from baseline amounted to 26% in the ®rst year increasing to 69% by the ®fth. These ®ndings are critically examined in terms of both internal and external validity. Future research will focus on di•erential e•ects of speci®c elements of the behavior-based safety initiative described herein. # 1999 Elsevier Science Ltd. All rights reserved.

Keywords: Behavior-based; Industrial safety; Accident prevention process

1. Introduction

Too often, the researcher and practitioner work in isolation from one another, in spite of the value of blending their roles. The practitioner often comes across valu- able knowledge useful to the researcher, yet it is rare that a consultant ®nds a way to

*Corresponding author. Tel.: +1-805-646-0166; fax: +1-805-646-0328; e-mail:[email protected]

0925-7535/99/$ – see front matter # 1999 Elsevier Science Ltd. All rights reserved. PII: S0925-7535(99)00007-7

2 T.R. Krause et al. / Safety Science 32 (1999) 1±18

conduct research related to their own services. Non-research directed missions, dif- ®culties with data collection in the ®eld, and the pressure to make business decisions before long-term research become available all discourage practitioners from con- ducting outcome research. The present study was a safety consulting organization's e•ort to systematically evaluate the results of its clients.1

Behavioral approaches to safety performance improvement have enjoyed increas- ing recognition as e•ective solutions to occupational health and safety challenges. Some of the most frequently cited components of behavior-based safety are goal- setting and posted feedback (e.g. Chhokar and Wallin, 1984; Reber et al., 1984), observation and posted feedback (e.g. Komaki et al., 1980), and observation, verbal feedback, data analysis, and problem solving (e.g. Krause et al., 1990; Krause, 1995). McAfee and Winn (1989) described the content of various behavior modi- ®cation programs in greater detail and summarized 24 published studies. Guastello (1993) conducted a meta-analysis of seven studies on the e•ectiveness of behavior- based safety programs. The common features of these programs were employee training, observation, and feedback. However, they di•ered in their use of goal- setting, incentives, method of observation, type and method of feedback, and in their longevity. In the present approach, front line employees of the implementing organization were trained to conduct observations, provide feedback, analyze behav- ioral data, and use behavioral data to make system improvements.

2. Background of employee-driven, behavior-based safety

Komaki et al. (1978) were among the ®rst researchers to apply behavioral tech- niques to industrial safety when they worked with a food manufacturing plant. In their study, operational de®nitions of safety-related behavior were developed and used to observe and record behavior on a checklist. The experimenters provided posted and verbal feedback to workers 3±4 times weekly, based on data gathered during observations. Results showed improvements in safe behavior of 26% in one department and 21% in another. When the observation program was withdrawn, performance returned to baseline. This 25-week project did not examine the e•ects of the initiative on injury rates. The pioneering research of Komaki et al. (1978) demonstrated that applied behav-

ior analysis methods could be used e•ectively to improve safety-related behavior in the workplace and, at the same time, raised several questions. If safety-related behaviors improved, would comparable change in incident rates follow? How could organizations learn to implement and maintain behavioral methods themselves without dependence on outside consultants? How can site personnel sustain perfor- mance improvement? Who should conduct observations and give feedback? What types of feedback work best? To what degree would ®ndings of concomitant changes

1 While the speci®c technology described and evaluated in this study is owned by a company for whom its authors work, every e•ort has been made to conduct and report this evaluation without bias. Never- theless, readers should be aware of this fact when assessing the ®ndings.

3 T.R. Krause et al. / Safety Science 32 (1999) 1±18

in incident frequency and safe behavior generalize to other behaviors not directly measured? Subsequent research has addressed some of these questions and neglected others.

That speci®c knowledge of results is important was shown by Locke et al. (1981) and that speci®c feedback facilitates development of a more detailed plan for behavior change (Kopelman, 1986). Kopelman also reported that when feedback revealed a performance de®cit, it was important to develop an action plan to take corrective action. Earley (1988) demonstrated that action planning mediated the in¯uence of speci®c feedback. Speci®c feedback stimulated more detailed planning than general feedback and led to a higher level of performance. While behavioral scientists learned about feedback and its relationship to goal-

setting and action planning, organizational theorists were developing methods for managing quality. Deming's (1986) work on management systems and their e•ect on quality transferred directly to industrial safety management. Krause et al. (1990) point out that much of the observed variation in injury rates is attributable to ran- dom variation. They, as well as others in the ®eld, recognized that when statistical process control charts indicate that incident rates are in control (i.e. most observed variation is random), system changes are called for if incident rates are expected to improve (Krause and Hidley, 1989; Salazar, 1989; Krause, 1995). Deming (1986) held staunchly to the view that it is management's responsibility to

change the system. Krause and Hidley (1989) combined this approach with applied behavior analysis and targeted safety as an improvement area. Management con- trols training resources, develops and implements policies and procedures, regulates spending for equipment, and selects and places personnel. When management understands its responsibility for employee safety and directs improvement e•orts on the safety system it created, then management also understands that blaming the employee will not result in safety improvements. Deming (in quality) and Krause and Hidley (in safety) prescribed adequate training, measurement of upstream pro- cess indicators, in-depth assessments, and positive feedback on the status of the system. At the same time, involvement of employees at all levels was recognized as the mechanism for continuous improvement in safety. Taken together, these developments in organizational behavior management,

combined with methods from total quality, suggested some important possibilities for industrial accident prevention. Krause (1989, 1992) con®rmed Komaki et al.'s (1978) ®nding that the most e•ective observation strategy is to target site-speci®c behaviors that have led to incidents in the past. Rather than using a standard list of behaviors that may or may not be applicable in a particular setting, Krause and colleagues adapted quality improvement techniques into standard methods for identifying relevant behaviors from incident reports (Krause et al., 1984, 1990; Killimett, 1991a). They developed a method for providing immediate positive verbal feedback following observation (Krause et al., 1990; Killimett, 1992; Krause, 1997). Krause and his colleagues involved employees at all levels of the organization in

order to develop a self-sustaining mechanism for continuous improvement. This approach replaces the traditional model of training supervisors and managers to observe behavior and deliver feedback, or having outsiders perform this function.

4 T.R. Krause et al. / Safety Science 32 (1999) 1±18

Employees across levels learned to observe and give feedback, and then to use behavioral data to select safety improvement targets. Managers and supervisors play an important role in removing barriers to safe

performance and facilitating the smooth operation of the process. Because em- ployees implemented the safety initiative, they possessed the tools to train new participants and were personally invested in the improvement e•ort (Killimett, 1991b; Krause, 1995). This approach de®nes the phrase `employee-driven, behavior- based safety process'. Most studies of the e•ectiveness of behavior-based safety programs, including

those cited by McAfee and Winn (1989) and Guastello (1993), have been pre- dominately time-limited, lasting weeks and months rather than years. These studies rely heavily on external resources for implementation and did not provide a sus- taining mechanism. To date, a systematic and long-term evaluation of the e•ect of behavior-based

safety with observation, feedback, data analysis, and action planning components has not been reported. The present study is the ®rst in a series of process and out- come evaluations. It adds to the existing literature a longitudinal evaluation of an employee-driven accident prevention process implemented in real-world industrial settings.

3. Materials and methods

3.1. Participants

Participants were 73 facilities who responded to requests for data, from a popu- lation of 229 facilities who met the inclusion criteria for this study. Three inclusion criteria reduced the 407 sites using the employee-driven behavioral accident preven- tion down to the target population of 229. First, each steering committee must have been trained by an outside consultant, as opposed to having been trained by an employee of their company (internal consultant). The internal consultant model of implementation uses a di•erent implementation strategy than the one being investi- gated here and is the subject of a separate study. Each site also had to be located in North America and had to have completed at least 1 year of observations. This long-term independent variable was necessary to allow adequate time to a•ect the global and long-term outcome measure, incident rate. All sites were asked to send monthly, non-cumulative recordable cases and hours

worked, covering only those employees potentially a•ected by the behavioral pro- cess, beginning 4 years before behavioral observations began. To give an idea of how much data was being requested, a site beginning observations in 1987 would have been asked to compile as many as 336 numbers from as many separate records. Considering that it was typical for a site never to have compiled their monthly injury data, this was a substantial undertaking. Moreover, for organizations not previously inclined to gather and analyze monthly incident data, the task of motivating a response for this project was even more di†cult. Sites gave the following types of

5 T.R. Krause et al. / Safety Science 32 (1999) 1±18

reasons for not responding: “The data are stored in another location and I don't have access to it'', “I don't know where to ®nd that data'', “It would take too much time to pull all that together'', “I only have summary reports'', “I can't get the monthly hours worked from the personnel department'', “We changed our record- ing criteria and didn't apply the new criteria retroactively'', and “We've had a major reorganization, so it's impossible to get injury data for the group of employees doing the behavior-based safety process''. Because only 73 of the 229 target sites provided data, selection bias was a concern.

To better understand the extent of selection bias, the 73 participants were compared to the 156 non-participants on several variables: longevity of the process, number of employees, union representation, industry, and responses to the question “Do you view your (behavior-based safety) process as a success?''2 Means and percentages of these variables for each group are shown in Table 1. Independent groups t-tests assuming unequal variances revealed that the participants extended their behavior- based safety process to more employees (M=533) on the average than non- participants (M=363), and that this di•erence was signi®cant (t (110)=2.60, p<0.01). This ®nding suggests that larger facilities may have been able to devote resources to data gathering. Similarly, participants had newer processes, or fewer years of observation (M=3.11 years) than non-participants (M=4.15 years). This di•erence in longevity was also signi®cant (t (122)=3.56, p<0.01). A chi-square test showed that the distribution of sites across industries did not di•er between partici- pants and non-participants, and binomial tests showed that the groups did not di•er in being represented by a union or not. Finally, the groups did not di•er in response to the question “Do you view your process as a success?'' This is an important

Table 1 Comparison of participants to non-participants

Participants Non-participants

M or % n M or % n

Number of employees participating* 533 73 363 156 Longevity (years since observations began)* 3.11 73 4.15 156 Union representation 48% 73 57% 156

Industry 73 156 Chemical 30% 22 27% 42 Paper 21% 15 15% 23 Petroleum 12% 9 15% 23 Other 37% 27 43% 68

“Do you view your process a success?'' (Yes) 97% 58 91% 92

*Two-tailed p<0.01.

2 As part of a separate and unpublished research project, the employees responsible for coordinating the behavior-based safety processes (the facilitators) at 150 sites who were also members of the target population in the present study were interviewed. One of the questions was “Do you view your process as a success?''

6 T.R. Krause et al. / Safety Science 32 (1999) 1±18

non-indication of selection bias: if strong selection bias were operative we would have expected to ®nd di•erences in response to this question. Because of signi®cant di•erences between participants and non-participants on

the two variables, size and longevity of process, post hoc analyses were conducted to determine the likely impact of these di•erences on the e•ect size estimates.

4. Procedure

4.1. Design

This study utilized an interrupted time-series design with 73 replications. A meta- analysis combined results at individual facilities for the larger study. At each loca- tion studied, the independent variable was the behavior-based safety initiative, described in the following section, and the dependent variables were the United States of America Occupational Safety and Health Administration (OSHA) record- able injury/illness rates. In the US, OSHA recordable cases are “All work-related deaths and illnesses, and those work-related injuries which result in: Loss of consciousness, restriction of work or motion, transfer to another job, or require medical treatment beyond ®rst aid'' (US Department of Labor, 1986, p. 17). Recordable rates were computed by multiplying the number of recordable injuries in a given period by 200,000 and dividing by the number of hours worked in the same period. This is roughly equivalent to the number of recordable cases per 100 employees per year.

4.2. Independent variable

All 73 facilities received training for a behavioral science approach to accident prevention originally described in Krause et al. (1990) and later in Krause (1997). Behavior-based safety reduces incidents through management of at-risk behaviors. This approach, which identi®es, measures, and improves safety-related behaviors, is based on measurement, upstream sampling of key variables, feedback, problem solv- ing, and employee involvement. It identi®es and corrects existing systems that pro- duce at-risk behavior and develops new systems that encourage safe behavior. Although each behavior-based safety initiative was unique, they shared several

important features. At each site, a safety assessment was conducted. The consultant evaluated existing strengths and weaknesses via direct observation, interviews, sur- veys, and analysis of incident data. The consultant's written assessment report to the organization included a recommendation for the implementation plan. Following each company's assessment, at least one steering committee was

formed. The steering committee would be trained to become the driving force behind the initiative. This group acquires the knowledge and develops the skills and experience required to implement behavior-based safety. This built-in mecha- nism kept the processes functioning over the long-term without constant input from an outside consultant. Committee membership was approximately 80%

7 T.R. Krause et al. / Safety Science 32 (1999) 1±18

non-management employees and 20% ®rst-level supervisors. The chairpersons were usually non-management employees. Each committee's ®rst tasks were to develop a checklist of critical behaviors and

write operational de®nitions for each behavior. Each implementation was unique in that the number and de®nitions of critical behaviors varied depending on the behav- iors that had contributed to incidents in the past. For example, two sites identi®ed “Use of Barriers and Guards'' as critical behaviors. The steering committee at one chemical company de®ned it as: “Barricades are placed around all jobs where entry by an unsuspecting person could result in an injury'', e.g. “Barricades shall be placed just outside the swing radius of TEX 554, 555, and 557''. The steering com- mittee at an electric company developed a slightly di•erent de®nition: “Is the area/ equipment that is at risk properly identi®ed to make people aware of the danger?'' For example: “When grating is removed, does the person install red barricade tape? Does person install “Danger High Voltage'' sign when leaving cubicle open with voltage present?'' Each site studied had developed their critical behaviors checklist and de®nitions from an analysis of their incident reports whereby they extracted behaviors that had contributed to incidents in the past and de®ned them using examples that made sense for their unique situation. Once the checklist was developed, the steering committees selected and trained

observers to use the checklist to observe behavior and then to give feedback. When presenting the feedback, the observers learned to begin by describing the observed behaviors which were completed safely (success feedback). Next, they learned to describe the at-risk behaviors they observed, and discuss these with the employee(s) (guidance feedback). They learned to use this discussion in a positive way to discover exactly what happened that was at-risk and why. Typically, observer training lasted 2 days. It included education on the foundations of behavior-based safety, information on how the critical behaviors checklist and de®nitions were developed, classroom training on how to identify at-risk behavior (usually slides depicting safe and at-risk behaviors), classroom training on interaction skills, and ®eld training. During the ®eld training, new observers would conduct observations under the guidance of a steering committee member who served as a coach. Finally, the steering committees learned to use observation data to identify and

solve systemic problems. After each observation, the scores from the data sheets were entered into a database anonymously (so as not to identify individual em- ployees). Typically, steering committees printed monthly reports showing, for example, the percent safe score for each behavior. Employees then had the oppor- tunity to select high-risk behaviors for problem solving. On the average, the steering committee training described above took place over

9 months for each site. Because feedback is one of the key mechanisms to improve safety performance in a behavior-based safety initiative, the date observations began was used as the start date for each project. Observations typically began in the 6th month of implementation. In the present study, baseline was de®ned as the 4-year period prior to the date observations began, while the follow-up period began immediately after observations began and extended through to the present.

8 T.R. Krause et al. / Safety Science 32 (1999) 1±18

4.3. Materials

Each steering committee received training manuals, slides, and a software package for data collection.

4.4. Data collection procedures

In June 1993, a task force was formed for the purpose of data collection on this project. The e•ort began with a letter requesting monthly non-cumulative US OSHA recordable rates beginning 4 years before implementation (hereafter referred to as `results data'), in exchange for a statistical analysis. Although the same results data were requested from each organization and every e•ort was made to gather the complete results set (multiple phone calls, providing forms, discovering why data were not sent and removing barriers wherever possible), sites were encouraged to submit whatever data they could retrieve from their records. Beginning in 1996, all target organizations were also o•ered complimentary participation in a behavior- based safety benchmarking service3 in exchange for the anonymous use of the data for research. Of the 229 sites that met the inclusion criteria described above, 120 provided

OSHA recordable injury/illness rates4: a 52% response rate. Of these, 47 were excluded because they failed to provide enough data to permit statistical tests. Table 2 provides details on the inadequacies of these data. It shows the minimum requirements for monthly and annual data, and the frequency with which sites failed to provide the minimum by period (baseline, follow-up, or both).5 This left 73 sites, or 32% of the target population, to include in the study. Nineteen of the 73 partici- pants had sent annual data, four sent quarterly, while 50 sites sent monthly US OSHA recordable rates. Thirty-three of the participants shared their data through the benchmarking service, while the remaining 40 participants shared their data for other reasons (e.g. response to authors' request, custom analysis).

Table 2 Frequency of failures to send adequate data by type of data and period

Period

Type of data Minimum requirement Baseline Follow-up Both

Monthly Annual

12 months per period 2 years per period

2 2

6 0

28 9

3 This is a complimentary service available to the entire population of interest to this study. Sites use the benchmarking service, which produces quarterly benchmarking reports, to compare process and out- come measures, to identify best practices in behavior-based safety, and to network with one another.

4 Two sites provided their best estimates of the OSHA recordable injury/illness rates. Of these, one site did not report to OSHA but provided a near equivalent based on medical treatment, lost time, and restricted duty cases. The other site provided recordable rates based on disabling injuries/illnesses alone.

5 Adequate data are continuously solicited from sites and incorporated into the meta-analysis. The most current results are available from the authors.

9 T.R. Krause et al. / Safety Science 32 (1999) 1±18

5. Results

Prior to analysis, each data set was entered and veri®ed for accuracy of data entry. Data were also screened for inconsistencies. For example, if a site provided both annual and monthly injury counts, the monthly numbers were aggregated and the results compared to the annual ®gures. All discrepancies were reconciled by con- tacting the site and verifying the data. Three types of analyses were conducted. A meta-analysis determined the average

e•ect size of the intervention strategy. Individual time-series analyses provided supporting detail. Post-hoc analyses were used to evaluate the extent of selection bias. The meta-analysis compared the percent reduction in injuries over baseline in each

of the ®rst, second, third, fourth, and ®fth years after observations began. A paired t-test comparing baseline levels (M=8.49) to post-initiative levels after 1 year of observations (M=6.24) showed that, on the whole, the 73 sites achieved signi®cant reductions in injuries (t (72)=7.31, p<0.0001). Table 3 shows that the average reductions from baseline were 26% in the ®rst year, 42% from baseline in the sec- ond, 50% in the third, 60% in the fourth, and 69% in the ®fth year after observa- tions began. Note that the percentages are calculated sequentially over baseline. For example, the Year 2 percent reduction was calculated from the recordable rate in Year 2, not from the recordable rate for Years 1 and 2 combined. Individual time-series analyses provided detail on each site's safety performance.

Preliminary analyses were conducted to identify seasonality, trends, and other autocorrelations in the data. Trends were of particular concern because they fre- quently appear in time-series injury data. Therefore, all data, even if annual, were tested for trends using standard regression analysis. Because of the large impact of identi®able trends on the subsequent time-series

analysis, there was concern over the regression analyses' accuracy in detecting trends in the annual data. Although power is increased by the smaller variance in aggre- gated data, power is decreased by the availability of fewer degrees of freedom. To measure the impact of aggregating data on the ability to detect trends, each of the monthly data sets where trends had been detected in the baseline were aggregated into annual data sets and the trend test was repeated on the annual baseline data. Of the 14 monthly data sets with one or more years of baseline, trends were detected in the aggregated data in 11 cases (79%). This con®rms that the annual data are less desirable than the monthly data, but that they usually do not mask trends. The quantity of information in the annual data was deemed useful, accurate, and sensi- tive enough to warrant inclusion in the study. All data were tested for trends, and monthly data were tested for seasonality. In 56

cases, there was no evidence of serial dependence (trends or seasonality) in the data, which meant independent groups t-tests were appropriate. In 16 cases, a pre-existing trend required the use of a t-test of the residuals from the pre-initiative regression lines. The trends were detected in a regression analysis of the recordable rates over time using a two-tailed signi®cance level of p<0.05. Seasonality was detected in one case using an analysis of autocorrelations with a two-tailed signi®cance level of p<0.05. This site's

1 0 Table 3

Demographics, results for individual projects, and overall results

Site Industry DOB a No. of Variable BL BL Sequential Percent employees tested years rate b follow-up rate reduction from baseline

Y1 Y2 Y3 Y4 Y5 Y1 Y2 Y3 Y4 Y5 t (df)

1 Chem 01-86 500 RIR 2.00 6.00 5.64 4.68 1.22 1.87 6 22 80 69 2.40 (3)** 2 Chem 06-88 1000 RIR 2.42 5.88 3.06 2.44 1.57 1.21 1.11 48 59 73 79 81 4.43 (4)**** 3 Chem 01-89 1100 RIR 1.00 7.77 4.04 1.68 48 78 3.26 (25)*** 4 5

Chem Chem

04-89 11-89

700 442

RIR RIR

1.25 1.84

15.32 4.54

17.00 3.31

10.42 2.07

7.58 3.39

8.08 4.51 3.60

ÿ11 27

32 54

51 25

47 1 21

3.05 (71)*** 1.60 (68)*

6 Chem 10-90 500 TS 3.75 8.30 4.39 3.89 4.64 1.74 3.11 47 53 44 79 63 2.57 (81)*** 7 Chem 11-90 1000 RIR 3.17 20.95 14.98 14.80 28 29 3.85 (54)**** 8 Chem 09-91 400 RIR 4.00 12.65 8.54 5.77 3.16 2.06 32 54 75 84 3.33 (6)*** 9 Chem 09-91 98 RIR 3.67 9.43 5.66 0.00 8.48 40 100 10 1.92 (5)* 10 Chem 11-91 186 RIR 3.08 3.18 1.27 2.26 2.36 60 29 26 1.34 (70)* 11 Chem 01-92 70 RIR 2.00 9.81 4.52 5.42 1.70 4.42 0.93 54 45 83 55 91 2.84 (27)*** 12 Chem 09-92 485 RIR 2.67 1.60 0.39 0.58 1.34 0.57 0.43 76 64 16 64 73 2.64 (44)*** 13 14

Chem Chem

09-92 01-93

1125 1800

RIR RIR

4.00 3.00

3.52 5.16

4.28 2.80

1.86 1.88

2.27 2.13

1.72 ÿ22 46

47 64

36 59

51 3.52 (94)*** 5.50 (5)****

15 16 17 18

Chem Chem Chem Chem

06-94 11-94 02-95 03-95

540 700 625 51

RIR REG RIR RIR

3.00 4.00 4.00 3.94

6.72 3.80 7.22 3.60

6.83 1.97 7.28 2.04

3.62 1.65 4.73

3.25 ÿ2 48 ÿ1 43

46 57 34

52 1.89 (74)** ÿ1.19 (62) 1.01 (21)* 0.75 (19)

19 Chem 05-95 550 REG 4.00 5.63 3.69 3.91 34 31 4.93 (72)**** 20 21

Chem Chem

11-95 04-96

93 450

RIR RIR

4.00 4.00

6.10 2.48

6.45 2.42

ÿ6 2

ÿ0.43 (63) 0.08 (58)

22 Chem 05-96 335 RIR 4.00 4.28 4.17 3 0.73 (33)

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.