Assume that you are leading a staff development meeting o

Original Article

Evaluating the Impact of EBP Education: Development of a Modified Fresno Test for Acute Care Nursing Margo A. Halm, PhD, RN, NEA-BC

Keywords

modified Fresno, EBP education/ competencies,

acute care nursing, novice-to-expert,

psychometrics

ABSTRACT Background: Proficiency in evidence-based practice (EBP) is essential for relevant research find- ings to be integrated into clinical care when congruent with patient preferences. Few valid and reliable tools are available to evaluate the effectiveness of educational programs in advancing EBP attitudes, knowledge, skills, or behaviors, and ongoing competency. The Fresno test is one objective method to evaluate EBP knowledge and skills; however, the original and modified versions were validated with family physicians, physical therapists, and speech and language therapists.

Aims: To adapt the Modified Fresno-Acute Care Nursing test and develop a psychometrically sound tool for use in academic and practice settings.

Methods: In Phase 1, modified Fresno (Tilson, 2010) items were adapted for acute care nursing. In Phase 2, content validity was established with an expert panel. Content validity indices (I-CVI) ranged from .75 to 1.0. Scale CVI was .95%. A cross-sectional convenience sample of acute care nurses (n = 90) in novice, master, and expert cohorts completed the Modified Fresno-Acute Care Nursing test administered electronically via SurveyMonkey.

Findings: Total scores were significantly different between training levels (p < .0001). Novice nurses scored significantly lower than master or expert nurses, but differences were not found between the latter cohorts. Total score reliability was acceptable: (interrater [ICC (2, 1)]) = .88. Cronbach’s alpha was 0.70. Psychometric properties of most modified items were satis- factory; however, six require further revision and testing to meet acceptable standards.

Linking Evidence to Action: The Modified Fresno-Acute Care Nursing test is a 14-item test for objectively assessing EBP knowledge and skills of acute care nurses. While preliminary psycho- metric properties for this new EBP knowledge measure for acute care nursing are promising, further validation of some of the items and scoring rubric is needed.

INTRODUCTION Over a decade ago, the Institute of Medicine (Institute of Medicine [IOM], 2001) recognized evidence-based practice EBP as a key solution to ensure care delivered has the high- est clinical effectiveness known to science. To reach the IOM’s (2007, p. ix) 2020 goal that “90% of clinical decisions will be supported by accurate, timely and up-to-date clinical informa- tion that reflects the best available evidence,” nurses need EBP competencies to guarantee that relevant research findings are integrated into clinical situations when congruent with patient preferences (Melnyk, Gallagher-Ford, Long, Long, & Fineout- Overholt, 2014).

BACKGROUND A recent evidence synthesis reported 10 studies evaluating the effectiveness of educational interventions in building EBP attitudes, knowledge, skills, and behaviors of nurses (Halm,

2014). Interventions were primarily workshop or immersion programs, but seminars, journal clubs, and EBP and research councils were also evaluated via: (a) self-reported EBP attitude, knowledge, and behavior (Chang et al., 2013; Dizon, Somers, & Kumar, 2012; Edward & Mills, 2013; Leung, Trevana, & Waters, 2014); (b) PICO questions and activity diaries (Dizon et al., 2012); (c) Edmonton Research Orientation (Gardner, Smyth, Renison, Cann, & Vicary, 2012) and Clinical Effectiveness or EBP Questionnaire (Sciarra, 2011; Toole, Stichler, Ecoff, & Kath, 2013; White-Williams et al., 2013); and (d) interviews and focus groups to identify qualitative themes about nurses’ expe- rience in EBP programs (Balakas, Sparks, Steurer, & Bryant, 2013; Nesbitt, 2013; Wendler, Samuelson, Taft, & Eldridge, 2011). Varied measurement across studies limited estimation of the effectiveness of EBP training (Dizon et al., 2012).

In a systematic review, Shaneyfelt et al. (2006) rec- ommended valid and responsive methods to evaluate the programmatic impact of EBP education and progression in

272 Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International

CE

EBP competencies. As self-report is extremely biased (Lai & Teng, 2011; Shaneyfelt et al., 2006); objective knowledge tests that incorporate multiple-choice or short answers with case-based decision-making like the Berlin Questionnaire (Fritsche, Greenhalgh, Falck-Ytter, Neumayer, & Kunz, 2002) or Fresno test were recommended to evaluate EBP knowledge and skills (Shaneyfelt et al., 2006). The Fresno test, a valid and reliable method to evaluate EBP knowledge and skills using a standardized scoring rubric, has been validated with family physicians (Ramos et al., 2003), physical therapy (Miller, Cummings, & Tomlinson, 2013; Tilson, 2010), and speech language (Spek, de Wolf, van Dijk, & Lucas, 2012).

SPECIFIC AIMS As objective methods for assessing EBP knowledge and skills of nurses are lacking, the specific aim of this study was to fill a measurement gap by adapting the modified Fresno test (Tilson, 2010) for acute care nursing. Only with consistent use of psy- chometrically sound methods can useful evidence be generated about the effectiveness of various EBP teaching strategies— new knowledge that can direct effective educational and pro- fessional development programs for students and practicing nurses. The specific research question was: Will an adapted Fresno test discriminate EBP knowledge and skills between novice, master, and expert acute care nurses?

METHODS Research Design A cross-sectional cohort design was used to replicate Tilson’s (2010) modified Fresno test (Figure 1).

Phase I: Test adaptation. New scenarios on acute care nurs- ing were developed for items #1–8 that remained unchanged. Item #9 (clinical expertise) was retained despite removal due to poor psychometric performance by Tilson (2010). Items #10–13 were modified for acute care although the EBP focus was un- changed. Item #14 was modified to the best design for studying the meaning of experience.

Phase 2: Content validity. Content validity was established with a panel of four masters and doctorally prepared acute care EBP experts from practice and academic settings. In round one, panelists rated each item and rubric for clarity, impor- tance, and comprehensiveness on a 5-point Likert scale. Pan- elists provided feedback on whether items should be retained, revised, dropped, or added (Polit & Beck, 2012). In round two, items #10 (mathematical calculations for sensitivity, positive predictive value) and #11 (relative and absolute risk reduction) were replaced because the panel did not believe acute care nurses would be expected to make these calculations without a resource. These items were replaced (and reviewed) with assessing tool reliability/validity and applying qualitative find- ings. The scoring rubric (Figure S1) was modified to reflect item alterations and ensure scoring consistency across subjects and raters (Jonsson & Svingby, 2007). With a single overall score,

Figure 1. Study flowchart.

a passing score was defined as >50% of available points for in- dividual items (Tilson, 2010). This passing score was set lower than that defined as “mastery of material” (Ramos, Schafer, & Tracz, 2003) to reduce the risk of a floor effect with novices.

A content validity index (I-CVI) was calculated for individ- ual items by dividing the number of 4–5 ratings by the number of experts. Mean (M) item ratings were 4.54 (clarity), 4.82 (im- portance), and 4.75 (comprehensiveness). Only item 12 had an I-CVI value <0.78 because the panel rated interpreting con- fidence intervals lower on importance for acute care nurses. The scale CVI of .95% was calculated by averaging I-CVIs, exceeding acceptable standards of >.90 (Polit & Beck, 2007; Table 1).

Phase 3: Validation of modified Fresno. After Institu- tional Review Board exemption was obtained, invitations were emailed to three cohorts: (a) novice nurses (less than 2 years of

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International

273

Original Article Table 1. Modified Fresno Test Items (n = 90)

Scores

Item/EBP step or component Topic

Content validity index (I-CVI)

Possible score

Passing score

Novices (n= 30) M (SD)

Masters (n= 30) M (SD)

Experts (n= 30) M (SD) p value*

1 INQUIRE PICO question .92 0–24 >12 13.73 (7.37) 19.47 (3.71) 18.13 (4.55) .001 (N-M, N-E)

2 ACQUIRE Sources 1.0 0–24 >12 15.03 (6.53) 20.33 (5.09) 17.53 (6.05) .004 (N-M)

3 APPRAISE Treatment design

1.0 0–24 >12 5.80 (6.77) 10.50 (6.90) 11.90 (5.87) .001 (N-M, N-E)

4 ACQUIRE Search .92 0–24 >12 13.93 (5.06) 16.53 (4.69) 15.10 (4.69) .18

5 APPRAISE Relevance .92 0–24 >12 7.47 (6.31) 9.77 (6.83) 12.03 (6.72) .03 (N-E)

6 APPRAISE Validity .92 0–24 >12 7.30 (6.75) 10.67 (7.77) 10.23 (7.38) .16

7 APPRAISE Significance 1.0 0–24 >12 3.40 (3.94) 9.97 (8.18) 7.70 (7.03) .001 (N-M, N-E)

8 PATIENT PREFERENCES

Patient preference

1.0 0–16 >8 6.13 (4.36) 8.20 (5.59) 9.00 (4.95) .08

9 CLINICAL EXPERTISE

Clinical expertise

1.0 0–8 >4 4.80 (3.04) 5.60 (2.49) 6.40 (2.49) .08

10 APPLY Tools .92 0–12 >6 3.90 (4.18) 8.50 (3.35) 7.00 (4.12) .001 (N-M, N-E)

11 APPLY Qualitative 1.0 0–16 >8 12.13 (4.75) 10.93 (5.35) 12.53 (6.19) .50

12 APPRAISE Confidence intervals

.75 0–4 >2 .13 (.73) .40 (1.22) 1.07 (1.80) .02 (N-E)

13 APPRAISE Design diagnosis

1.0 0–4 >2 .27 (1.01) .27 (1.01) .27 (1.01) 1.00

14 APPRAISE Design meaning

1.0 0–4 >2 2.13 (2.03) 3.73 (1.01) 3.87 (.73) .001 (N-M, N-E)

Total scores .95 Scale CVI 0–232 >116 96.17 (26.14) 134.87 (30.76) 132.77 (28.94) .001 (N-M, N-E)

*Scheffe post-hoc analysis: N= Novices; M= Masters; E= Experts.

experience after graduation from a bachelorette program) from three U.S. Magnet hospitals; (b) master nurses (master’s pre- pared) recruited via the National Association of Clinical Nurse Specialists listserv; and (c) expert nurses (doctorally prepared) recruited via the American Nurses Credentialing Corporation’s Magnet program director’s listserv and faculty at Bethel Uni- versity (St. Paul, MN, USA). Nurses in the expert cohort self- affirmed their EBP expertise and teaching experience. Up to 1 hr (in one sitting) was allowed to complete the test with no external resources; only notepaper and calculators were per- mitted. Reminder e-mails were sent at 2 and 4 weeks. A $10 gift certificate incentive was offered upon completion. Some participants did not answer all the items on the exam; these participants were not included in the sample for each cohort. Only participants who had a complete exam were included in the analysis. Data were collected in 2015.

Two doctorally prepared nurses with expertise teaching EBP served as raters after an orientation to the test items and scor-

ing rubric. Raters practiced scoring three pilot tests from the three cohorts and resolved discrepancies that could threaten in- terrater reliability (IRR; e.g., halo effect, leniency or stringency, central tendency errors; Castorr et al., 1990; before scoring commenced. A midway refresher session allowed raters to re- view scores, reducing the threat of rater drift (Castorr et al., 1990). Data were analyzed with SPSS Version 23.0 (IBM Corp., Armonk, NY, USA).

RESULTS Descriptive Statistics The total sample of 90 nurses included cohort (a) new grad- uates (n = 30); (b) master’s prepared CNSs (n = 30); and (c) doctorally prepared nurses (n = 30). Seventy-six percent completed the test within 60 min (83% novices, 70% mas- ters, 73% experts). Mean min for test completion were 56.43

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International

274

Development of a Modifi ed Fresno Test for Acute Care Nursing

Table 2. Psychometric Properties of Individual Items (n = 90)

% Passed by cohort

Item # Topic ICC IDI ITC All

(n= 90) Novices (n= 30)

Masters (n= 30)

Experts (n= 30) χ2 p-value

1 PICO question .78 .43 .53 85.6 63.3 100.0 93.3 18.52 .0001

2 Sources .78 .35 .53 84.4 73.3 93.3 86.7 4.74 .09

3 Treatment design .86 .61 .56 44.4 26.7 50.0 56.7 6.03 .05

4 Search .72 .26 .48 80.0 76.7 86.7 76.7 1.25 .54

5 Relevance .48 .65 .63 35.6 26.7 33.3 46.7 2.72 .26

6 Validity .47 .43 .50 32.2 20.0 43.3 33.3 3.76 .15

7 Significance .74 .52 .57 26.7 6.7 40.0 33.3 9.55 .01

8 Patient preference

.55 .52 .39 52.2 36.7 50.0 70.0 6.77 .03

9 Clinical expertise .23 .22 .40 88.9 80.0 93.3 93.3 3.60 .17

10 Tools .76 .74 .68 68.9 40.0 90.0 76.7 18.77 <.0001

11 Qualitative .68 .17 .31 88.9 93.3 90.0 83.3 1.58 .46

12 Confidence intervals

.90 .04 .12 13.3 3.3 10.0 26.7 7.50 .02

13 Design diagnosis .61 .13 .12 6.7 6.7 6.7 6.7 .00 1.0000

14 Design meaning .89 .35 .37 81.1 53.3 93.3 96.7 22.77 <.0001

Total score .88 N/A N/A .0001

(standard deviation [SD] 38.21) for novices; 57.20 (SD 42.54) for masters; and 43.21 (SD 26.33) for experts.

Reliability Statistics IRR was calculated using intraclass correlation coefficients (ICC) for total score and individual items (Table 2). Total score reliability was high at .88. Of the 14 items, 3 had excellent reliability (>.80), 7 had moderate reliability (.60–.79), and 4 had questionable reliability (<.60). Items with questionable IRR focused on relevance (#5), validity (#6), patient preference (#8), and clinical expertise (#9). A Cronbach’s alpha coefficient of .70 was obtained for internal consistency of the modified exam.

Item discrimination index (IDI) was calculated for each item by separating total scores into quartiles and subtracting the pro- portion of nurses in the bottom quartile who passed that item (>50% points per item was passing) from the proportion in the top quartile who passed the same item. The 50% threshold has been defined as “mastery of material” (Ramos et al., 2003) and used in similar validation studies (Tilson, 2010). IDI ranges from –1.0 to 1.0, representing the difference in passing rate between nurses with high (top 25%) and low (bottom 25%)

overall scores. Eleven of the 14 items had acceptable IDIs >.2 (Table 2). Correlation between item and total score and cor- rected item-total correlation (ITC) was assessed using Pearson correlation coefficients. Twelve of the 14 items had acceptable ITCs >.3 (Table 2). Low IDI and ITC items focused on con- fidence intervals (#12) and design for diagnostic tests (#13). Qualitative findings (#11) also had a low IDI.

Total Score Analysis No floor or ceiling effect was apparent, indicating the test is ap- plicable from novice to expert (Figure 2). As shown in Table 1, total mean scores for novices (M 96.17, SD 26.14) revealed that a passing score of 116 was not achieved in this cohort as with the master (M 134.87, SD 30.76) and expert (M 132.71, SD 28.94) cohorts. One-way analysis of variance (ANOVA) demonstrated that overall mean scores were significantly dif- ferent, F (2, 89) = 17.58, p < .0001, between cohorts. A post- hoc Scheffe comparison showed novice total mean scores (M 96.17, SD 26.14) differed significantly from master (M 134.87, SD 30.07, d = 1.36) and expert nurses (M 132.77, SD 28.94, d = 1.33). Cohen’s d is an effect size measure that is used to explain the standardized difference between two means,

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International

275

Original Article

Figure 2. Box plots for sum scores.

commonly reported with ANOVAs or t tests. There were no significant differences between the master and expert cohorts.

Item Score Comparison Post-hoc Scheffe analysis also revealed significant cohort dif- ferences in eight items (Table 1). Novice nurses scored sig- nificantly lower than master and expert nurses on PICO (#1), sources (#2), treatment design (#3), relevance (#5), significance (#7), tools (#10), confidence intervals (#12), and design mean- ing (#14). On the other hand, the mean scores for four items increased progressively across cohorts from novice to master, and then from master to expert. These items were treatment design (#3), relevance (#5), patient preference (#8), and con- fidence intervals (#12). While not all items performed in this manner, these items demonstrated mastery of EBP material across cohorts.

Item Difficulty Item difficulty (IDI) was calculated via the proportion of nurses who achieved a passing score for each item (Table 2). Of the 14 items, none were easy (IDI > .8). Ten items (71%) were moderate (IDI > .3), and 4 (29%) were difficult (IDI < .3; Janda, 1998; Nunnally & Bernstein, 1994). In testing individual items, all three cohorts scored below the passing cutoff for five items: Treatment design (#3), validity (#6), significance (#7), confidence intervals (#12), and diagnosis design (#13). Novice and master nurses did not achieve a passing score for relevance (#5), while novices did not pass patient preferences (#8) and tools (#10).

Using chi-square analysis, seven items showed significant differences in the proportion of passing scores between cohorts (Table 2). Masters scored highest on PICO (#1), significance (#7), and tools (#10). Experts performed best on treatment de- sign (#3), design meaning (#14), patient preferences (#8), and confidence intervals (#12).

In examining item discrimination based on the propor- tion of nurses who passed the test (Table 2), some significant items did not discriminate well between masters and experts: (a) PICO (#1); (b) treatment design (#3); (c) significance (#7); and (d) design meaning (#14). Items on sources (#3), search (#4), relevance (#5), validity (#6), and expertise (#9) discrim- inated on the IDI but did not assess unique EBP knowledge and skills among the three cohorts (p > .05).

DISCUSSION The Modified Fresno-Acute Care Nursing test is a 14-item test for assessing EBP knowledge and skills. While the original test assessed core principles of EBP steps, this replication val- idated patient preferences and clinical expertise to fully assess all EBP domains. The test has excellent content validity with I-CVIs ranging from .75 to 1.0. Overall scale CVI was .95. In- ternal consistency was acceptable at .70. Table 3 compares the psychometric properties of the Modified Fresno-Acute Care Nursing test with the original and modified tests.

Total scale reliability for the two independent raters was excellent (.88). IRR for individual items was good to excellent for 10 of 14 items (71%). One reason IRR may have been lower for relevance (#5) and validity (#6) was the rubric complexity that required raters to consider responses for both items when scoring. Like Tilson (2010), IRR was less than desirable for pa- tient preference (#8) and clinical expertise (#9). Some leniency in scoring may have occurred with #8 when a nurse offered a phrase that could elicit patient preferences, rather than stating it as a question as specified in the rubric. As recommended by Tilson (2010), clinical expertise should be retained as it covers an essential EBP domain, but further revision and validation is needed.

Item difficulty was moderate to high. Two items retained from Tilson’s (2010) version had low IDI and ITC: Confidence intervals (#12) and design for diagnosis (#13). These items were difficult across cohorts and did not discriminate. Of the new items, tools (#10) had acceptable psychometrics across ICC, IDI, and ITC. The second qualitative item (#11) had accept- able ICC and ITC but low IDI and did not discriminate across cohorts. This finding may demonstrate that qualitative find- ings have a rich tradition of emphasis across levels of nursing education and practice.

While some items did not perform ideally, these items re- main valuable to the larger research goal of developing an objective and responsive method to evaluate EBP knowledge and skills. Reasons for poor item performance may include item characteristics, unknown sample characteristics, scoring concerns, or a combination of these factors. Six items (#5, #6, #9, #11, #12, and #13) need to be revised and retested before be- ing removed. Although Tilson (2010) dropped clinical expertise (#9), it covers an important EBP domain that other researchers recognized as essential for measurement (Miller et al., 2013).

A range in item difficulty is best so that the high and low range of ability can be evaluated. For item #12 (confidence

Worldviews on Evidence-Based Nursing, 2018; 15:4, 272–280. © 2018 Sigma Theta Tau International

276

Development of a Modifi ed Fresno Test for Acute Care Nursing

Table 3. Comparison of Reliability and Validity of Fresno Tests

Performance

Measure/acceptable results

Original Fresno (Ramos et al., 2003)

Dutch adapted Fresno (Spek et al., 2012)

Modified Fresno-physical therapy (Tilson, 2010)

Modified Fresno-Acute Care Nursing test (Halm, 2018

current study)

Population � Family physicians � Speech language, clinical epidemiology students

� Physical therapy � Acute care nurses

Total score/# items � 212/12 � 212/12 � 224/13 � 232/14

Content validity

� Scale CVI/>.90 � Not reported � .92 � Not reported � .95

Interrater reliability

� Interrater correlation/

� >.60

� Items: .72–.96 � Total score: .97

� Not reported � Total score: .99

� Items: .41–.99 � Total score: .91

� Items: .23–.90 � Total score: .88

Internal reliability

� Cronbach’s/>.70 � Item-total correlation (ITCs)/>.30

� .88 � .47–.75 (items)

� .83 � .31–.76

� .78 � .20–.66

� .70 � .12–.68

Item discrimination </

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.