In our second discussion in this unit, we will look at the 360 evaluation. As we have explored the 360 evaluation, we have l

The Use of 360-Degree feeDback compareD To TraDiTional evalUaTive feeDback

for The professional growTh of Teachers in k–12 eDUcaTion

abstract

Empirical research on the use of 360-degree feedback in elemen- tary and secondary educational settings is quite limited. This study sought to understand teachers’ perceptions of the quality of feedback they received from the traditional administrative evaluative feedback to feedback they received from a multi-source feedback process. Results from the Wilcoxon Matched-Pairs Signed Ranks Test indicated that the participants in this project found the multi-source feedback process to be significantly more helpful than the traditional method in a number of areas, including: the development of professional growth goals, identifying professional devel- opment needs and providing feedback focused on student achievement.

The research and literature on the teacher evaluation process for more than three decades criticizes current teacher evaluation methods (Thomas, 1979; Scriven, 1981; McGreal, 1983; Prybolo, 1998; Peterson, 2000; Asel- tine, Faryniarz, & Rigazio-Digilio, 2006; Toch & Rothman, 2008). It has been suggested that school systems need to evaluate their teacher evalua- tion processes in order to bring them into alignment with their mission, vi- sion, values and goals as well as to provide a meaningful exercise for both principals and teachers. Holland and Garman (2001) suggest that there is little to no evidence supporting the claims that evaluative supervisory visits to classrooms support instructional improvement. They question the legiti- macy of supervisory visits being a professional practice for improvement of instruction or a legally mandated practice for evaluation of teaching. Reeves (2006) also warns leaders that “effective and ineffective practices are the re- sult not of random chance, but of deliberate choice” (p. 166). In a study con- ducted by Kersten and Israel (2005), the current summative evaluation pro- cess in use was found to be too vague or too generic to be of any substantive use. Comments from administrators in their study reflect the lack of effec- tiveness of the traditional summative process: • The evaluation system is out of date and has not changed in decades; • The system is not comprehensive enough to have any real impact; • The criteria for ratings were inadequately defined and inconsistently

interpreted;

Jo-Anne Mahar Barbara Strobert

Planning and Changing Vol. 41, No. 3/4, 2010, pp. 147–160

147

• Although a district-wide process is in place, it does not yield any meaningful feedback for teachers [italics added]. (p. 58)

Danielson and McGreal (2000) identified six main deficiencies in current teacher evaluation systems: a) they utilize outdated, limited, evaluative criteria; b) they indicate few shared values and assumptions about good teaching; c) they lack precision in evaluating performance; d) they are hierarchical one-way communication; e) there is no difference between novice and experienced practitioners; and f) they are conducted with limited administrator expertise. As school districts develop commit- tees to improve their teacher evaluation systems, Danielson and McGreal recommend they focus their discussions on: 1. Those practices that are realistic for the district in terms of teacher and

administrator time demands. 2. The availability of resources to support the training necessary to make

new systems function effectively. 3. The level of commitment that the administration, the board of educa-

tion, and the teachers union have to break away from more traditional views of evaluation. (p. 17)

Danielson and McGreal go on to mention the potential benefits of feedback from parents, teachers, and colleagues in this process. In a study conducted in Nevada, Sawyer (2001) found that teachers complained that there was no collaboration and their evaluation process provided them with very little productive feedback. King (2003) suggested that K–12 school districts need to break away from traditional views of teacher evaluation by reviewing the research and literature and providing teachers with more options within this process. Included in her recommendations are such op- tions as professional growth plans based on measurable goals, peer review and coaching, professional portfolios and the use of 360-degree feedback process in the development of professional growth goals.

The idea that teachers’ performance evaluations should be built upon multiple data sources is supported in Koçak (2006), Douglas & Douglas (2006), Marzano (2003), Danielson & McGreal (2000), and Gray-Smith (2000). Danielson and McGreal recognize the value of 360- degree feedback systems as a data collection option in the teacher evalua- tion process. The 360-degree feedback is a contemporary feedback strat- egy focused on building professional growth. The 360-degree feedback procedure relies upon feedback from peers, subordinates, supervisors, and others within the evaluatee’s circle of involvement. The intent is to link feedback received to the organization’s goals and initiatives and to the em- ployee’s professional career development.

For 20 years, research has shown that 360-degree feedback can enhance communications and performance when the employee is held ac- countable to develop a professional growth plan in line with the organiza-

Mahar Strobert

Planning and Changing148

tion’s mission, vision, values, and goals (Bernadin & Beatty, 1987). Schools might also experience similar results if the 360-degree feedback process is offered as an option for the annual performance review of teachers. This study explores the possibility that K–12 teachers can also benefit from re- ceiving feedback from more than one source, which has traditionally been the administrator. This might include feedback from multiple sources such as students, parents, and colleagues as well as the administrator.

Student ratings of teachers show the strongest positive relationship to student achievement when compared with those of principals and teach- ers (Wilkerson, 1997; Manatt & Kemis, 1997; Manatt & Benway, 1998; Koçak, 2006). This is also corroborated by Peterson (2000), who found that data from student surveys and questionnaires can be highly reliable due to the large numbers of students as reporters. Manatt suggested that if the feedback process is done right, it can be the foundation for school trans- formation efforts.

There is very little research on the use of the 360-degree feedback process in K–12 public education. There have been several recommenda- tions suggesting further study of this process in schools, as it may be an- other viable option for districts to consider for the professional growth of teachers toward student achievement. This study tested this theory by rep- licating previous studies conducted at the Research Institute for Studies in Education (RISE) at Iowa State University.

methods and procedures

This descriptive study utilized a non-experimental, quantitative and qualitative survey design to compare teachers’ experiences with the tradi- tional single-source feedback obtained from the summative annual perfor- mance review process to the feedback obtained from the 360-degree feed- back process. A qualitative analysis was conducted by analyzing the open ended statements included on all surveys. A quantitative analysis was con- ducted based upon the responses obtained from the pre- and post-study sur- veys completed by teachers participating in this pilot project. The research design and methodology had several limitations and delimitations: a) this study was confined to one suburban school district in the state of New York, therefore the results could not be generalized to any other group; b) teacher survey responses were representative of their individual experiences with traditional evaluative feedback compared to their personal experiences with the 360-degree feedback process. As with many survey studies, the return rate limited the amount of feedback obtained for teacher use.

Sample

This pilot project took place in a large suburban district in the cen- tral Hudson Valley region of New York State, which is representative of

The Use of 360-Degree Feedback

Vol. 41, No. 3/4, 2010, pp. 147–160 149

other New York State suburban school districts. In this district, there are nine elementary schools, three middle schools and one high school, with a total population of slightly more than 10,000 students and 786 teachers currently employed in grades K–12. Twelve initial volunteers were select- ed by a random sampling method. A table of random numbers was used (Witte & Witte, 2007) to identify the first 12 participants from the district data base of personnel. This was stratified by gender and grade level. After the first 12 participants completed the process, 15 additional participants volunteered to participate. A total of 27 teachers from grades K–12 com- pleted the process during the 2007–2008 school year. Table 1 shows the percentage of participants by assignment area.

Table 1

Percentage of Participants by Assignment Area

Grade level n Percentage K–2 6 22.2 3–5 8 29.6 6–8 7 25.9 9–12 4 14.8 Othera 2 7.4

Total 27 99.9

Note. This percentage has been rounded and therefore the total is less than 100%. aOther teaching staff participating included special education, remedial, or speech teachers.

Instrumentation

All surveys and the resulting data sets obtained for this study were provided by the Research Institute for Studies in Education (RISE) at Iowa State University. Participating teachers completed electronic pre- and post-study questionnaires comparing 360-degree feedback, the indepen- dent variable, with traditional single-source evaluative feedback (Gray- Smith, 2000). All surveys utilized a Likert scale for responses: 1 = Strongly Agree, 2 = Agree, 3 = Disagree, 4 = Strongly Disagree. Teachers respond- ed to 14 statements that were focused on obtaining their perceptions of the quality of feedback they receive from the current, single source evaluation method (pre-study survey) and the quality of feedback they receive from the 360-degree method (post-study survey). These dependent variables in- cluded fourteen identical statements on both pre- and post-study surveys that sought to elicit participants’ responses indicating how each system provides: a) feedback on the effective performance of job responsibilities; b) feedback that promotes professional growth; c) feedback to guide future professional development; d) reports that are practical for the improve-

Mahar Strobert

Planning and Changing150

ment of job performance. One additional statement was added to the post- study survey: “the 360-degree process enhances the traditional system.”

This project also utilized Teacher-to-Teacher feedback question- naires containing 13 items, Parent-to-Teacher feedback questionnaires con- taining 25 items, and Student-to-Teacher feedback questionnaires contain- ing 20 items. The majority of the items adopted for use on the feedback instruments were selected from the pool of valid, reliable, and legally dis- criminating items identified in previous studies. The findings of previous research conducted by refining the survey questionnaires were accepted for the purpose of this study (Omatoni, 1992; Weber, 1992; Wilcox, 1995; and Wilkerson, 1997).

Procedures

The pre-study survey for participating teachers was posted elec- tronically using surveymonkey.com. Participants were given a seven day window to complete the survey.

Parents were chosen randomly from each teacher’s student roster and were sent confidential survey packets including the Parent-to-Teacher survey and a self addressed, stamped envelope for return. Students of partic- ipating teachers completed the Student-to-Teacher surveys under the super- vision of district support staff (guidance counselors, social workers or school psychologists) after obtaining parental permission. Participating teachers were not present during student survey administration. Colleagues of par- ticipating teachers were randomly chosen and sent a confidential Teacher- to-Teacher survey packet for completion. Parents and colleagues were given a 10-day window to complete and return the surveys. All returned survey packets were then sent to the RISE consultant at Iowa State University for data processing. The participating teachers were provided with the parent, student, and colleague feedback results in a booklet format from the RISE consultant. All results were completely confidential and provided only to the participating teachers. Participants were given a two week time period to reflect upon these results before completing a post-study electronic sur- vey based on their experiences with the 360-degree feedback process.

Data Analysis

Analysis was performed using SPSS for Windows Version 17.0. Since the population sample size was relatively small (n = 27), the Wil- coxon Signed Rank Test was used to determine if the difference between the pre-study questionnaire and post-study questionnaire scores are great- er than would occur by chance: Z = the smaller of R+ or R-. The Wilcoxon signed-rank test is a non-parametric test of two related samples on a single sample. It is an appropriate measure when the dependent variables are or- dinal and samples are related as in this study.

The Use of 360-Degree Feedback

Vol. 41, No. 3/4, 2010, pp. 147–160 151

A T-test for Paired Samples also provided a comparison of differ- ences between the pre- and post-measurements. This test was performed to compare the means from the survey results of the traditional method to the survey results of the 360-degree feedback method and to determine if there is a significant difference between the means, or if the differences are due to sampling errors created by random sampling.

findings

The purpose of this study was to gain knowledge of the use of the 360-degree feedback method in K–12 education and to determine if this method of feedback is effective in assisting teachers when developing pro- fessional growth goals and identifying professional development needs. Results from this 360-degree feedback pilot study were analyzed to an- swer the following research questions: • To what extent does the traditional single-source feedback method of

evaluation provide useful feedback to teachers? • To what extent does the 360-degree feedback model provide useful

feedback to teachers? • To what extent does the 360-degree feedback model compare to the

traditional single-source feedback model in assisting teachers in de- veloping professional growth goals?

• To what extent does the 360-degree feedback model compare to the traditional single-source feedback model in assisting teachers in de- veloping professional development needs?

The Traditional Method

There were no positive comments offered regarding the type of feedback received from the traditional method from the pre-study survey. The teacher participants reported that the traditional method is not effec- tive in providing feedback related to their daily work. Respondents indi- cated that a shortcoming of the traditional method is that it provides little guidance toward instructional improvement, and administrator feedback is vague. They find the processes to have little relevance or impact on their educational practice, and they receive many irrelevant and positive evaluations. In other words, the participants in this study agree with Peter- son (2000) that the vast majority of teachers find the feedback from tradi- tional observation checklists less than professionally meaningful, and they agree with Aseltine et al. (2006), who also contend that this method rarely helps teachers make a direct link with their professional growth and stu- dent learning needs. Most disconcerting is that only 29.6% of the partici- pants find the traditional method to be focused on student achievement.

Mahar Strobert

Planning and Changing152

The 360-degree Feedback Method

The most frequent complaint about the 360-degree process from participants was the lack of responses from colleagues and parents, which was indicated in Tables 2 and 3.

Table 2

Teacher-to-Teacher Feedback by Grade Level

Grade level n Sent Returned % returned K–2 6 48 35 73.0 3–5 8 64 63 98.0 6–8 7 56 20 36.0 9–12 4 32 19 59.0 Othera 2 16 15 94.0

Total 27 216 152 70.4 aOther teaching staff participating included special education, remedial, or speech teachers.

Table 3

Parent-to-Teacher Feedback by Grade Level

Grade level n Sent Returned % returned K–2 6 60 48 80.0 3–5 8 80 70 88.0 6–8 7 70 20 29.0 9–12 4 40 17 43.0 Othera 2 16 12 75.0

Total 27 260 167 64.2 aOther teaching staff participating included special education, remedial, or speech teachers.

Another issue that surfaced was that some of the survey statements were inappropriate for a particular respondent, for example: “My teacher gives appropriate homework.” Some teachers do not give homework; therefore they believe this statement not to be helpful. A final criticism was the fact that having an outside agency involved with the process made the process slower than might have occurred if surveys were not sent out of district for processing.

Several positive comments about the 360-degree process were noted. Respondents generally liked the process and found that the feed- back contained in the final report was quite helpful, especially the student survey results. This supports the findings of Peterson, Wahlquist, Bone, Thompson and Chatterton (2001) and Wilkerson, Manatt, Rogers, and

The Use of 360-Degree Feedback

Vol. 41, No. 3/4, 2010, pp. 147–160 153

Maughan (2000) claiming that data from student surveys and question- naires can be highly reliable due to the large numbers of students as re- porters, as indicated on Table 4. It was noted that the process offered some clear areas that can assist teachers with improvement.

Table 4

Student Feedback by Each Grade Level

Grade level Teacher participants Student participants K–2 6 117 3–5 8 208 6–8 7 106 9–12 4 85 Othera 2 13

Total 27 529 Percentage total student return 100

aOther teaching staff participating included special education, remedial, or speech teachers.

Comparisons

Table 5 presents the means, t-scores, degrees of freedom and sig- nificance level of each of the survey items.

Table 5

Paired Samples T-test

Feedback Criteria M t df 2-tail

significance 1a. The promotion of sound educational principles .230 1.296 25 .207 1b.The effective performance of job responsibilities .360 2.092 24 .047* 1c. The fulfillment of district and school goals .160 .891 24 .382 1d.The promotion of professional growth .423 2.518 25 .019* 2a. The identification of strengths and concerns -.192 -1.044 25 .306 2b.The performance of job responsibilities .269 1.659 25 .110 2c. The guide to future professional development .461 2.483 25 .020* 3a. The creation of reports for the improvement of

job performance .423 2.391 25 .025*

3b.The identification of strengths .120 .569 24 .574 3c. The identification of weaknesses .115 .647 25 .523 3d.The job responsibilities of the person evaluated .423 1.897 25 .069 4a. The focus on teacher behaviors .416 2.846 23 .009*

(continued)

Mahar Strobert

Planning and Changing154

Feedback Criteria M t df 2-tail

significance 4b.The focus on student behaviors .545 2.806 21 .011* 4c. The focus on student achievement .666 2.436 23 .023*

* Significant at the .050 level.

There was a significant difference between the means of the pre-study and post-study survey on eight of the fourteen survey statements. The 360-de- gree feedback model proved to be significantly better than the traditional model on the following criteria: 1. The effective performance of job responsibilities. 2. The promotion of professional growth. 3. The guide to future professional development. 4. The creation of reports for the improvement of job performance. 5. The job responsibilities of the person evaluated. 6. The focus on teacher behaviors. 7. The focus on student behaviors. 8. The focus on student achievement. The differences on the remaining six criteria were found not to be statisti- cally significant.

The Wilcoxon Matched-Pairs Signed-Ranks Test indicated the 360-degree feedback model provided participants with more actionable feedback than the traditional feedback model as indicated in Table 6.

Table 6

Wilcoxon Matched-Pairs Signed-Ranks Test

Feedback Criteria Mean rank

(traditional/360) Z-

score 2-tail

significance 1a.The promotion of sound educational

principles 7.5/9.1 -1.29 .196

1b.The effective performance of job respon- sibilities

6.5/7.1 -2.00 .046*

1c.The fulfillment of district and school goals 5.8/6.07 -.884 .377 1d.The promotion of professional growth 6.0/7.9 -2.29 .022* 2a.The identification of strengths and concerns 8.7/9.4 -1.03 .302 2b.The performance of job responsibilities 5.5/6.1 -1.60 .109 2c.The guide to future professional develop-

ment 8.5/10.5 -2.27 .023*

Table 5 (continued)

(continued)

The Use of 360-Degree Feedback

Vol. 41, No. 3/4, 2010, pp. 147–160 155

Feedback Criteria Mean rank

(traditional/360) Z-

score 2-tail

significance 3a.The creation of reports for the improve-

ment of job performance 7.0/9.0 -2.20 .027*

3b.The identification of strengths 6.4/6.5 -.566 .571 3c.The identification of weaknesses 8.2/7.8 -.646 .518 3d.The job responsibilities of the person

evaluated 8.8/7.7 -1.99 .046*

4a.The focus on teacher behaviors .00/4.0 -2.42 .015* 4b.The focus on student behaviors 4.5/6.9 -2.44 .015* 4c.The focus on student achievement 11.6/9.5 -2.01 .044*

* Significant at the .050 level.

The difference in all of these ranks is negative, which indicates that the post-test scores (360-degree feedback), are ranked higher than the pre-test scores (traditional single-source feedback model).

The 360-degree feedback method was found to be significantly more effective in providing feedback that promotes professional growth, z = -2.29, p = 022. This finding supports the work of Grey-Smith (2000), who found the multi-rater system provides more data than the traditional method on which to base performance improvement.

In the study conducted by Mabey (2001), respondents rated their experiences with 360-degree feedback as providing much more focused staff development suggestions than the traditional model. Mabey’s find- ings are supported in this study, since the 360-degree process was found to be significantly better in assisting teachers in identifying professional de- velopment needs, z = -2.27, p = .023.

Contrary to the traditional method, 66.7% of the participants found the 360-degree feedback process to be more focused on student achieve- ment. This statement was found to be statistically significant on the Wil- coxon Matched-Pairs Signed Ranks Test (z = -2.01, p = .044). In light of this finding, one might conclude that the feedback obtained from the sur- veys provided teachers with information to improve instruction and there- fore student achievement. Since the vast majority of the returned surveys were from students, one might conclude that student feedback has an im- pact on instruction. This data supports the statement by Wilkerson et al. (2000): “students provide more valid feedback than teachers if student achievement is the validity measure” (p.187). Each test is significant to the .050 level, indicating these results are not just due to chance at that level.

Participants in this project were asked if the 360-degree process enhances the traditional system, and 81.5% agree that this model enhances the single-source traditional evaluation system. The 360-degree feedback

Table 6 (continued)

Mahar Strobert

Planning and Changing156

process has been successful in other organizations, but little research to date is available to determine its effectiveness in K–12 education. The re- sults of this study suggest that a 360-degree feedback option may indeed enhance the teacher annual performance review process.

Discussion

The participants in this project found the multi-source feedback pro- cess to be significantly more helpful than the traditional method in a num- ber of areas, among them developing professional growth goals, identify- ing professional development needs, and focusing on student achievement. Certainly student achievement must be at the forefront of the annual perfor- mance review of teachers. Manatt and Kemis (1997) claimed the overarch- ing purpose of teacher performance evaluation is to improve performance. Based upon the outcomes of this study, the 360-degree feedback process is a viable option for consideration as school leaders look to expand their teacher evaluation options with improved student achievement as the goal.

recommendations for practice

First and foremost, universities must review education supervi- sion courses. Many of these courses are based on the old model of super- vision and do not include the study of options such as developing profes- sional growth plans, teacher portfolios, data collection and goal setting, peer review, 360-degree feedback or a triangulation of methods.

Districts seeking to improve the annual performance review of teachers should consider utilizing several data sources, including feedback from students, parents and colleagues. If the traditional observation check list is currently being used, it should be carefully reviewed and differentiated, based on years of service, instructional level and/or content area. It should also include a rubric for scoring to improve consistency between evaluators as well as to provide teachers with clear expectations for performance.

The traditional observation check list should not be the sole meth- od of teacher annual performance reviews. One observation per year is grossly inadequate to guide a teacher’s professional growth. A combina- tion of methods must be considered, and the 360-degree feedback process should be offered as an option. District level policy makers need to em- ploy several methods for the annual performance review of teachers; as previously suggested by Douglas and Douglas (2006), a triangulation of information must be sought to monitor and manage the quality of educa- tion. The 360-degree feedback method has shown promise as an alterna- tive. Evaluation methods that promote collegiality between teachers and their administrators, colleagues, students, and parents support a culture of learning and professional growth.

Public education policy makers need to take a serious look at the po-

The Use of 360-Degree Feedback

Vol. 41, No. 3/4, 2010, pp. 147–160 157

litical nature of school governance in relation to school culture and teacher unions. As pointed out in the Kersten and Israel (2005) studies, “these factors contribute to a culture that supports status quo and squashes risk-taking and innovation” (p. 61). It is a conflict of interest when teacher unions control the evaluation process, so districts need to exercise caution when negotiating the teacher evaluation process. Policies should call for methods that differentiate between non-tenured teachers and tenured teachers, as well as between con- tent area teachers and others. Districts must tie staff development directly to student data and professional growth plans for teachers. It is also essential that school district annual performance review policies describe how the dis- trict trains administrative staff in the use of good evaluation practices.

School districts might want to consider discontinuing the use of the word “evaluation,” which suggests a top down, non-collaborative ap- proach. “Performance review” and “annual professional growth” are among some of the alternatives to be considered. Based upon the out- comes of this study, recommendations for using the 360-degree feedback process as a collaborative option for the annual performance review of teachers include: 1. An outside agency is not necessary for the implementation of the 360-

degree process; however, their experience is helpful when using the process for the first time.

2. Teachers and administrators should confer about the survey questions to be included on all surveys.

3. The administrator should also complete the confidential survey as a colleague in the process.

4. Teachers and administrators should also confer to discuss professional growth goals and development needs. All goals should be SMART: spe- cific, measurable, actionable, realistic, timely (Tucker & Stronge, 2005); and the goals should be focused on student achievement.

5. Teachers and administrators should occasionally confer to determine progress toward goals.

6. The summative evaluation process should be minimized and the for- mative processes maximized.

conclusion

The findings from this study provide insight into the use of the 360-degree feedback process in the public school sector. The process has been shown to provide participating teachers with reports that assist them in developing professional growth plans, guiding professional develop- ment needs and feedback that focuses on student achievement. The bene- fits …