Chapter 6: Descriptive research design, mixed methods, and meta-analysis & Chapter 7: Epidemiology and Longitudinal Studies Read Chapters 6 & 7 Discussion # 3 Based on your course reading a

6 CRITICAL APPRAISAL OF EVIDENCE

KAREN M. VUCKOVIC AND KATHERINE A. MAKI

■ INTRODUCTION

Whether one is reading a research article to translate fi ndings into evidence-based practice (EBP) or as a building block for a proposed study, the process oft en begins with appraising a single research article. Advanced practice nurses frequently read evidence (research, guidelines, reviews) to answer questions related to diagnosis, therapeutic interventions, and prognosis of individual patients (Dale et al., 2019). One of the barriers to translating research fi ndings (and therefore providing evidence-based care or guideline-directed therapy) into practice is a lack of confi dence in one’s ability to read and interpret research fi ndings (Gray et al., 2017). Th e critical thinking skills nurses use in practice every day provide a foundation for developing the skill of reading and evaluating research.

Reading and evaluating literature is a critical skill in translating research into clinical prac- tice. Previous chapters have addressed how to locate and retrieve evidence; chapters that follow detail the various elements and designs of collecting evidence and the reporting of results through publications. Th e purpose of this chapter is to review the sections of a single research article and to provide an organized approach to reading and interpreting the strength and relevance of the information presented in a data-based article.

Th e critical appraisal of the evidence determines the strengths, weaknesses, and usefulness of the fi ndings for practice and future research. Using a published critical appraisal tool to evaluate the evidence guides the clinician through a comprehensive critique (Moralejo et al., 2017; Zuzelo, 2019). Clinicians must weigh the limitations and feasibility of the evidence with its strengths to evaluate if the evidence is usefulness to practice. One type of evidence clinicians appraise is a clinical research study. As a result, understanding the components of an article based on data and the important questions to consider while reading each section is needed.

All research articles are written with a standardized format. Components of an article based on data include the title, abstract, background or introduction and signifi cance of the study, methods, data analysis, fi ndings and results, discussion, limitations, and implications for practice. Minor variations in the formatting may be required by a journal for publication, for example, limita- tions may be included in the discussion. Even so, the logic of the researcher’s thinking should be clear enough so that the reader has few questions about how and why the study was conducted. By the end of the article, the reader should be able to determine how the research results fi t into current knowledge and how (or whether) the fi ndings translate to the practice environment for implementation or necessitate further testing and validation. As the reader progresses through the article, each section builds on the previous information. Exhibit 6.1 summarizes key elements

Copyright Springer Publishing Company. All Rights Reserved. From: Research for Advanced Practice Nurses: From Evidence to Practice DOI: 10.1891/9780826151339.0006

108 II: Building Blocks for Evidence

EXHIBIT 6.1

EVALUATING A SINGLE RESEARCH ARTICLE—QUESTIONS TO CONSIDER

Abstract

Does the article fi t your research question or purpose? Practice setting? Population?

Background and Signifi cance

Literature Review Is the literature current and relevant? Is the research literature summarized and evaluated? Does the literature review introduce all concepts and variables proposed in the research? Are gaps in the literature noted? How likely is it that the current study will close the gaps in

current knowledge? Problem Statement/ Purpose (appears at the end of the Background section)

Are the aims of the study clearly stated? Is the research exploratory or hypothesis-testing? What is studied? What variables are measured (independent and dependent)? Does the purpose (or research question) clearly address the problem?

Method

Design What is the overall design of the study, quantitative or qualitative? Is the design a good match with the problem statement or purpose of the study?

Ethics How is the protection of human participants ensured?

Sample How is the sample identifi ed? Do the participants have characteristics that can answer the research

question? What are the inclusion and exclusion criteria? Instruments

Are the instruments used reliable and validated in the study population? Study Procedure

Is the procedure realistic? If an independent variable is manipulated, was it done so consistently? How are the instruments/ tools administered, in what environment, and was the environment

consistent? Over how much time are data collected?

Data Analysis

How are the data analyzed? Are statistical tests used appropriately? If the research is a qualitative study, how are the themes and meaning elicited?

Findings/ Results

What are the outcomes of the study? Are the results valid? Are all aspects of the problem statement/ purpose addressed? How do the fi ndings fi t with previous research? Are they supported or not supported?

(continued )

6 Critical Appraisal of Evidence 109

Discussion

What conclusions did the researcher draw from the fi ndings? Do the fi ndings make sense? Relate to the problem? How do the fi ndings compare with other research fi ndings in the literature? Can the results be generalized to other populations and/or settings? Will the results help me care for

my patients?

Limitations (may also be part of Discussion section) What limitations are noted? How will limitations affect generalizability of the fi ndings?

Conclusion/Implications

Are implications for practice and research noted? Do the conclusion/implications fl ow directly from the fi ndings?

to consider when appraising a data-based article and may serve as a general guide or checklist in reading the literature. Th is exhibit is one example of an appraisal tool; several other guides and tools are available including online soft ware to assist in critically appraising the data-based literature.

■ TITLE

Th e title describes and explains what the article is about. Th e title may include information about the focus or outcome of the research, the population studied, and the study design (Polit & Beck, 2020).

■ ABSTRACT

Th e fi rst part of a research article is the abstract. Th e abstract is a brief, targeted summary of the full article that follows. Th erefore a quality abstract presents a clear synopsis of the purpose, results, and research implications of the research manuscript. Th e abstract provides the reader with a succinct overview of the study and can be used to evaluate whether the study is of interest or applies to the reader’s practice setting or population (Alspach, 2017). Most readers use the abstract as a screen to determine whether or not to read the entire research article.

■ BACKGROUND (INTRODUCTION) AND SIGNIFICANCE

Th e background or introduction of a data-based article provides an overview of the current status of a specifi c fi eld and a context for the research. Th e fi rst few paragraphs provide the reader with an understanding of the background of the study, why the study was conducted, and why the study was important or signifi cant. Th e reader should be able to identify gaps in current knowl- edge and how the proposed study specifi cally fi lls the gaps. Near the end of the section (or set apart), the purpose (also referred to as the aim) or problem statement of the study is presented. Th e purpose or problem statement is closely related. Both or only one may be included in an arti- cle. Th e purpose or problem statement should be clearly stated and provide the independent and

110 II: Building Blocks for Evidence

dependent variables examined. Th e author may also include research questions, hypotheses tested in the study, or both. In any case, the reader will know the population (who) and the phenomenon (what) of interest. Th e reader uses this information to assess the remainder of the article.

■ LITERATURE REVIEW

In some cases, the background section includes a review of the literature. In other articles, the literature review is set apart as a separate section. Th e review of the literature should be appraised for both content and relevance. Th e literature presented should be relevant to the current study, relate to the variables that were studied, and be up to date. Th e literature review oft en includes reviews, theoretical and data-based sources. Th e previous research studies included in the back- ground section should at the minimum address the purpose, sample, design, fi ndings, and a brief critique of the study’s strengths and weaknesses (Gray et al., 2017). Another approach in report- ing the research literature is to review and synthesize numerous studies and evaluate the body of knowledge. Whichever approach is used, the reader should understand the existing knowledge and how the study may address gaps in knowledge or expand current knowledge. Th e research literature included in the review may be directly or indirectly related to the purpose of the study. Indirectly related studies should be linked for relevance.

Th e reader should check the publication dates of the literature cited and of the reference list to judge whether the references are (at least reasonably) current. Although some studies are con- sidered classics, much of the cited literature should be recent and refl ect up-to-date thinking and understanding of the study’s focus. Th is is especially important in practice areas undergoing rapid change (e.g., genetics and genomics) and in areas that are time sensitive (e.g., attitudes and opinions). Th e reader’s personal knowledge and level of expertise in the content area are valuable in determining the currency and strength of the literature review included in the research article.

■ METHODS

A large section of the research article is the methods section, which describes how the study was conducted. Th e methods section includes design, sampling, instruments, and specifi c procedures for data collection (Polit & Beck, 2020). Th e methods section is a critical part of a research article and deserves careful attention. While reading the methods section, the reader should be alert for any problems in the way the study protocol was implemented, such as sample bias, inconsisten- cies in data collection among participants, loss of participants or attrition, and weaknesses of the instruments or tools used to collect the data. Th e strength of the methods section helps the reader determine the overall usefulness and generalizability of the results that will follow.

Design Th e study design is identifi ed early in the methods section if it has not already been implied in the purpose or problem statement. Th e author should identify whether the study used a quantitative, qualitative, or mixed methods design. Quantitative studies use designs that result in numerical data that can be used in statistical (mathematical) analyses and assess the size of relationships among variables (Al-Jundi & Sakka, 2017). Variables in quantitative designs may be measured using physiologic instruments (e.g., blood pressure and weight), questionnaires with fi xed responses (e.g., scale of 1 to 5), or variables that can be assigned a number (e.g., age). Quantitative

6 Critical Appraisal of Evidence 111

research designs may be further identifi ed as experimental, quasi-experimental, or nonexperi- mental (descriptive or correlational), depending on how participants were chosen, whether and how study variables were manipulated, and how the data to measure the variables were collected (Gray et al., 2017).

Qualitative studies use a nonnumerical study approach to collect data, oft en to describe a phe- nomenon (Melnyk & Fineout-Overholt, 2019). Th e most common qualitative designs are ethno- graphic, phenomenological, historical, and grounded theory approaches. Just as in quantitative designs, there are specifi c and distinguishing elements among the qualitative designs. Th e goal of studies that use qualitative designs is to explore or explain the phenomenon of interest from the perspective of individuals experiencing the phenomenon. As a result, qualitative designs yield descriptions that can then be analyzed and coded for themes, common elements, and shared meaning among participants (Lewis, 2019). Th e end result of a qualitative design may be new knowledge or the beginning of a theory, whereas the end result of a quantitative study is oft en acceptance or rejection of current knowledge or theory.

In some instances, the study design may include both qualitative and quantitative elements to examine a specifi c research question or hypothesis, resulting in a mixed methods design. How methods are mixed varies greatly but three major issues need to be addressed in this type of design, that of timing, weighting, and mixing (Moorley & Cathala, 2019). Th e analysis of a mixed-methods study design combines both numeric and narrative data (Moorley & Cathala, 2019). Quantitative and qualitative design approaches can diff er greatly, with quantitative designs requiring large sample sizes and random selection or assignment to treatment groups, whereas some qualitative designs have small samples recruited from a narrow group of individuals. Th us, a mixed meth- ods study may have a large or small sample depending upon which research design dominates. Ultimately, sampling decisions are based on the research questions (Eckhardt & DeVon, 2017).

Th e design of the study should be suffi ciently detailed so that the reader can determine how the study was actually conducted. Th e timeline and sequence of the study procedures should be clear and concise so that the study can be replicated. Regardless of the overall research design, the key question for the reader to consider is how well the design used is likely to fulfi ll the purpose of the study and answer the research question.

Sample Th e number of participants who participated in the study or sample size should be clearly stated and described in the article including the number of participants who do not complete the study. In addition, the authors should describe how the sample size was determined. In a quantita- tive study, sample size is determined by power analysis, a mathematical determination based on the researcher’s desired level of statistical signifi cance, estimates of variability, and eff ect size (Al-Jundi & Sakka, 2017). A power analysis is not included in a pilot study because a primary purpose of a pilot study is to collect information to justify and guide subsequent, larger studies (Lowe, 2019). In many qualitative studies, the researcher describes how the (oft en small) sample size was suffi cient to answer the research question based on suffi cient “saturation” of information obtained (Malterud et al., 2016; Polit & Beck, 2020).

Th e article should also provide details regarding specifi c inclusion and exclusion criteria for participants to be enrolled as participants. Careful attention to potential participants excluded from the study will assist in determining whether fi ndings may be translated to clinical practice. Th e reader should be especially alert for any apparent bias in selecting the sample and exclusions that can limit generalizability of the fi ndings beyond the study’s individual sample and/ or setting.

112 II: Building Blocks for Evidence

A description of potential participants who were approached for inclusion but refused to par- ticipate should also be included to determine how closely the study population represents the population of interest as a whole.

Th e demographic and clinical characteristics of study participants (e.g., age, gender, comor- bidities) are usually presented and help the reader evaluate to what degree the study sample is congruent with the reader’s population of interest. Sample characteristics are also important in determining whether or not the study fi ndings might be applicable for translation into practice. Increased attention to include sex as a biological variable helps to prevent the generalization of fi ndings obtained from studies lacking power to test for sex diff erences between males and females (Lee, 2018). Th e more closely the sample matches the reader’s population, the more likely the reader is to implement the fi ndings into practice if all other criteria are met (no contradictory study results and other supporting studies with similar results). In addition, the number of partic- ipants and a brief description of participants who did not complete the study or study procedures should be included so that the reader can make a judgment whether individuals who completed the study were diff erent from those who did not complete the study, which is a potential source of bias called attrition (Kearney et al., 2017).

In quantitative studies, random selection of participants and random assignment of partici- pants to treatment groups are ideal but oft en diffi cult to accomplish because of the constraints that accompany research on human participants in a clinical environment. As a result, a conve- nience sampling method is oft en used and is strengthened when the design incorporates random assignment to treatment groups. Th e reader needs to make a judgment regarding bias in the sam- ple and the appropriateness of the sampling plan in answering the problem.

Research Instruments and Data Collection Tools Each research tool or instrument used in the study should be described in detail. Th e instruments should measure the variables of interest. If an existing tool or instrument was used (e.g., depres- sion scale), the number of items and a brief account of what the tool measures should be provided. Measures of reliability, the consistency of the tool or instrument, and validity (whether the tool actually measures the phenomenon under study) are important considerations and should be reported in the article (Vetter & Cubbin, 2019). It is imperative to include the reliability and valid- ity information the fi rst time a tool or instrument is implemented to study a specifi c population or demographic (Vetter & Cubbin, 2019). Th e choice of the specifi c tool used should be explained in the context of the study variables, previous research that used the tool(s), and any subject char- acteristics considered in choosing the tool (e.g., reading level and short administration time in a population likely to experience fatigue with a long tool). An advantage of using research instru- ments that have already been used is that reliability and validity data may already be established (Mayo, 2015; Polit, 2015). If the researcher had developed a tool for the study, a full description of the instrument and a discussion of how reliability and validity were established should be included. Whether an existing research instrument was used, or a tool was created for the study, a lack of information regarding reliability and validity leads to questions regarding whether or how well the variables in the study were actually measured (Vetter & Cuban, 2019).

Ethics Th e methods section should also include a short description of how ethical considerations in conducting the study were addressed. Alternatively, the protection of human participants may be

6 Critical Appraisal of Evidence 113

addressed as the fi rst part of the description of the study procedure. In either case, a statement regarding review of the study by an institutional review board or research ethics board prior to the beginning of the study is generally included (Al-Jundi & Sakka, 2016). In addition, procedures for obtaining participants’ consent to participate in the study and how consent was obtained should be detailed. If the participants were minors or were incompetent to provide informed consent personally, the author should fully describe assent (for minors) or consent procedures, mention whether any diffi culties were encountered, and, if diffi culties were encountered, how they were managed.

Study Procedures Th e procedure section provides a detailed description of how the study was conducted, including exactly how and when data were collected and under what conditions. Th e information should be clearly presented so that the reader could replicate the study by following the description. Th e reader should see a logical fl ow in the data collection process and consider any extraneous vari- ables in the setting that may aff ect the data.

■ DATA ANALYSIS

By the time, the reader comes to the data analysis section of the research article, the reader will know a great deal about the study. Th e reader has formed beginning opinions about the strength and potential usefulness of the study and is looking forward to reading the fi ndings and results. Th e data analysis section begins with a description of how the data obtained from the research instruments were summarized and analyzed. Th e intent of this section is to tell the reader how the data were analyzed and is a straightforward presentation of information. In a quantitative report, the data are analyzed using statistical methods and tests. Th ere are numerous statistical procedures and tests available. Th e key issue in evaluating the statistical analysis is to determine that the method used was appropriate for the research question and how the data were measured (Melnyk & Fineout-Overholt, 2019). In a qualitative study, the data analysis approach is described and coded, including how themes or patterns were elicited from the data. For the novice reader, the data analysis section may be the most intimidating part of the research article (Jeff eries et al., 2018). Th is discomfort is oft en due to limited exposure to and understanding of the statis- tical tests used and uncertainty about whether the appropriate test has been applied to the data. Resources that will aid the reader to develop their skills include a basic statistics book and col- leagues with an understanding of data analysis techniques. As with any skill, the more the reader gains in understanding, the easier reading the analysis section becomes.

■ FINDINGS AND RESULTS

For many, the most enjoyable section of a research article to read is the fi ndings of the study. Each previous section has been laying the foundation for this part of the article. Th e fi ndings tell the reader what the researcher discovered as a result of the data that were collected and analyzed. As the fi ndings of a study are presented, whether a qualitative or quantitative design was used, the reader learns whether the research question was answered and how completely the question or problem statement was addressed. All results and data that address the research question or problem statement are included in a discussion of the fi ndings. If the study was

114 II: Building Blocks for Evidence

analyzed using statistical methods, the statistical signifi cance (p-value or alpha) and confi dence intervals of the results are reported. It is important to note that p-values do not measure the eff ect size or clinical signifi cance of the results but convey the likelihood that the reported results were not achieved due to a false positive (or type I error) based on a predetermined alpha threshold, which is generally set to p < 0.05 (Wasserstein & Lazar, 2016). Th erefore, con- fi dence intervals can be more informative when interpreting results, especially when placed in a clinical context (Schober et al., 2018). In the results section, the data and outcomes of statis- tical analysis are presented but not explained or discussed. Th e intent of the results section is to present the factual outcome of data analysis, rather than explain the meaning of the data. Although this section may seem dry or unimaginative, the advantage of this approach is to allow the reader to make beginning judgments regarding the study outcomes in the absence of the opinion or interpretation of meaning from others. In addition to a narrative summary of the results, most articles present fi ndings using tables, graphs, or fi gures for easier review. It can be helpful to carefully review and interpret the data presented in the manuscript’s tables and fi gures before reading the results section in order to obtain a more comprehensive review of the data presented.

Qualitative study fi ndings, depending on the specifi c qualitative design used in the study, are presented quite diff erently from quantitative results. In a qualitative study, direct quotes or sum- maries of participant responses are oft en included in the results section or may be presented in a combined results/ discussion section (Cypress, 2019). Th e author may group the fi ndings accord- ing to themes or patterns that became apparent during the data analysis (Moorley & Cathala, 2019). As a result, many qualitative studies provide data using a narrative approach and describe results in terms of richness and depth of the data.

Some readers prefer to read the results section immediately aft er reading the problem or purpose of the study. Th is may be due to curiosity about the outcome or to decide whether or not to read the entire article. Th e dedicated reader will then go back to the beginning of the article and read it entirely. Th ere is nothing inherently wrong with reading the results out of sequence as long as the reader recalls that, in order to use the fi ndings in practice or to build additional research studies, the previous sections of the article are critical in evaluating the strength of the fi ndings. In addition, this approach may encourage a reader to fully read only those articles that report signifi cance or that reinforce current ways of thinking. Studies that do not demonstrate statistical signifi cance are oft en as revealing as those that do and encour- age us to challenge existing perceptions. Finally, because the results section presents but does not discuss the fi ndings, the reader may overlook studies with clinical (but not statistical) signifi cance.

■ DISCUSSION

In the discussion section of an article, the author presents the conclusions drawn from the fi nd- ings, acknowledges any limitations of the study, and suggests how fi ndings may be generalized to individuals or groups beyond the study sample. In the discussion, the author describes how the results fi t into the current body of general knowledge and specifi c previous research. Th e author should compare and contrast the study fi ndings with those of the previous research that was cited in the review of the literature presented earlier in the article. A critical comparison by the author demonstrates to the reader that the researcher evaluated the fi ndings with an open mind.

6 Critical Appraisal of Evidence 115

■ CONCLUSIONS

Th e author’s conclusions provide the researcher’s interpretation of the study fi ndings. In contrast to the factual presentation of the study outcomes in the fi ndings section, the conclusions present the meaning of the results from the author’s perspective. Th e conclusions drawn by the researcher should fl ow from the scope of the study and directly relate to the purpose of the study; they should be confi ned to the variables that were studied. Th e reader should evaluate the author’s perspective as well their own to determine whether the fi ndings answer questions in the reader’s clinical experience and/or provide information that explains phenomena previously unexplained.

■ LIMITATIONS

Th e author’s identifi cation of the study’s limitations recognizes that, although no study is perfect, the results can contribute and provide valuable information for future researchers (Melnyk & Fineout-Overholt, 2019). At the same time, limitations cannot be used as an excuse for a poor design or fl awed study procedures. Among the limitations oft en cited in research reports are problems with data collection (e.g., unexpected intervening variables that occurred during data collection), small sample size, problems with how the sample was obtained (e.g., convenience sample), and limitations inherent in the study’s research design (e.g., non-random assignment of participants to groups; Melnyk & Fineout-Overholt, 2019). In most cases, the reader has already identifi ed limitations and is not surprised by those noted by the author. Th e limitations will aff ect the reader’s confi dence in translating the fi ndings into clinical practice.

■ GENERALIZABILITY

Th e generalizability of study fi ndings is an essential evaluation of a study’s outcome. Studies are conducted with participants who have specifi c characteristics and in settings with unique environments. In addition, manipulation of the independent variable and measurement of the dependent variable may be done in more than one way, and researchers may use comparable or divergent research instruments or tools. As a result, the meaning of the fi ndings and how the fi ndings may be implemented with other populations and in other settings must be addressed in the article. An understanding of the limitations of the study also aff ects generalizability. A study with numerous or key limitations results in fi ndings that have minimal or narrow generalizability beyond the population or setting in the study. Th is is especially likely when bias is present in the sample. Bias may be a design fl aw or may be unintentional and discovered during data analysis.

■ IMPLICATIONS

Th e fi nal major aspect of a research article is its implications for practice and research. An import- ant goal of research is to

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.