Define synthesis Explain your understanding of the type of li

Educational Researcher, Vol. 46 No. 3, pp. 131 –139 DOI: 10.3102/0013189X17703946 © 2017 AERA. http://edr.aera.net

ApRIl 2017 131

Research syntheses in education, like meta-analyses (Glass, 1976) and best-evidence syntheses (Slavin, 1986), are conducted to identify evidence-based practices.1 They do

so by combining findings across empirical studies whose con- structs are sufficiently similar to warrant comparison, which makes constructs essential to framing research. As the building blocks of theory, constructs are characterized by how they link abstractions to observed phenomena given social, historical, political, and cultural assumptions at work in conceptualizing them (Watt & Van Den Berg, 2002). In research, constructs carry disciplinary assumptions about what effects matter and for whom (e.g., Eisenhart & DeHaan, 2005; Latour & Woolgar 1979/1986). For instance, familiar constructs studied in educa- tion, such as “high achieving,” “disabled,” and “career ready,” come preloaded with assumptions about students and what effects might matter for them.

Research syntheses are conducted in hermeneutic circles (Skrtic, 1991); the constructs included and excluded from them anticipate how problems will be framed and solutions formu- lated. For example, research syntheses that include medical accounts of the construct “disability” invoke medical solutions while syntheses that include social accounts of disability invoke social solutions. As such, answers returned to the research syn- thesis question, “What works to improve employment for peo- ple with disabilities?” will depend on how “disability” is first

formulated. Making explicit the social, political, historical, and/ or cultural accounts of “disability” can reveal the hermeneutic circle in which a research synthesis is involved and suggest pos- sibilities for including alternative accounts of constructs.

Although many narrative (Petticrew & Roberts, 2006) and sys- tematic review methods—such as metanarrative (Greenhalgh, Macfarlane, Bate, Kyriakidou, & Peacock, 2005), metastudy (Paterson, Thorne, Canam, & Jillings, 2001), and critical interpre- tive synthesis (CIS; Dixon-Woods et al., 2006)—can be drawn upon to critique constructs, none fully answers our call, through a systematic process, to understand how constructs and methodologi- cal elimination decisions frame the results of research syntheses. In Table 1, we clarify research synthesis as our object of critique. We characterize and differentiate research syntheses, such as meta- analysis and best-evidence synthesis, from other systematic reviews that do not include methodological elimination as an integral part of their study screening process. We also show how neither tradi- tional nor qualitative systematic review methods include a system- atic process to critically analyze constructs in existing research syntheses. Interrogating constructs in research synthesis is needed (we argue) in the field of education, where governments and educa- tionalists privilege these syntheses to answer questions about what

703946 EDRXXX10.3102/0013189X17703946Educational ResearcherEducational Researcher research-article2017

1University of South Florida, Tampa, FL 2University of Kansas, Lawrence, KS

Unpacking Assumptions in Research Synthesis: A Critical Construct Synthesis Approach Jennifer R. Wolgemuth1, Tyler Hicks2, and Vonzell Agosto1

Research syntheses in education, particularly meta-analyses and best-evidence syntheses, identify evidence-based practices by combining findings across studies whose constructs are similar enough to warrant comparison. Yet constructs come preloaded with social, historical, political, and cultural assumptions that anticipate how research problems are framed and solutions formulated. The information research syntheses provide is therefore incomplete when the assumptions underlying constructs are not critically understood. We describe and demonstrate a new systematic review method, critical construct synthesis (CCS), to unpack assumptions in research synthesis and to show how other framings of educational problems are made possible when the constructs excluded through methodological elimination decisions are taken into consideration.

Keywords: critical theory; educational policy; disability studies; meta-analysis; qualitative research

FEATURE ARTIClES

132 EDUCATIONAl RESEARCHER

works in policy and practice (Donmoyer, 2012; National Research Council, 2002). To support dialogue and decision making that is more fully informed, contextualized, and critical, inquiry must also examine the social, historical, political, and/or cultural assumptions underlying constructs.

In this article we describe and demonstrate a new systematic review method, critical construct synthesis (hereafter CCS). A CCS explores and critiques constructs included and excluded from research synthesis, particularly through the process of screening out studies that fail to meet methodological standards for providing best

or quality evidence. It shows how reexamining research syntheses in light of constructs identified in methodologically excluded literature may open up possibilities for reframing educational problems.

Exclusion, Constructs, and Critique in Research Syntheses

In broad terms, the aim of any research synthesis is to summarize and evaluate research and knowledge on a topic. With the intro- duction of meta-analysis, research syntheses became tools for

Table 1 Types and Characteristics of Literature Reviews

Type of Literature Review Description

Systematic Process for Reviewing

Primary Literature

Includes All Primary

Literature

Includes Critical Analysis of Constructs

Critiques Existing Research

Synthesis

Traditional review A review that provides an overview of literature on a topic. Does not use systematic review methods.

No Yes No No

Critical review A review of literature that critically examines primary literature. Does not use systematic review methods.

No Yes Yes No

Systematic review Research synthesis:

Best-evidence synthesis

A systematic review that synthesizes only studies that meet predetermined methodological standards

Yes No No No

Research synthesis: Meta-analysis

A systematic review that summarizes studies through statistical comparison of findings

Yes Noa No No

Research synthesis: Narrative review

A systematic review that summarizes studies narratively, rather than by meta-analysis

Yes Noa No No

Metanarrative A systematic review that seeks to show various ways researchers have understood a heterogeneous topic area, often over time.

Yes Yes No No

Metastudy A systematic review that aims to generate new insights into phenomena through an analysis of theory, methods, and finding of qualitative research.

Yes Nob Yes No

Critical interpretive synthesis

A systematic review that aims to develop an interpretive model of a phenomenon from existing literature.

Yes Noc Yes No

Critical construct synthesis

A systematic review that aims to critique the constructs in literature included and excluded from an existing research synthesis.

Yes Yes Yes Yes

Note. Literature reviews relevant to the object of critique and methods in this article. For different and more exhaustive typologies of literature views, see, for example, Petticrew and Roberts (2009); Gough, Thomas, and Oliver (2012); and Kastner, Antonya, Soohiaha, Strausa, and Triccoa (2016). aSome meta-analyses and narrative reviews do not exclude quantitative studies based on the quality of their designs (Cooper & Hedges, 2009). They do, however, exclude qualitative and conceptual literature. bMetastudy does not exclude qualitative studies based on the quality of their designs (Paterson, Thorne, Canam, & Jillings, 2001). They do, however, exclude quantitative and conceptual literature. cCritical interpretive synthesis includes quantitative and qualitative studies but excludes those deemed “fatally flawed” (Dixon-Woods et al., 2006).

ApRIl 2017 133

quantifying an intervention’s effect by pooling estimates across stud- ies (Glass, 1976). As a quality control measure, reviews often restrict research syntheses to studies passing exacting standards of method- ological rigor. The U.S. Department of Education’s What Works Clearinghouse (WWC; 2014), for example, reviews only well-designed quasi-experimental studies, single-case designs, and randomized experiments. Yet, excluding studies not up to method- ological par conflicts with the idea of reviewing the full knowledge base. Glass (2000), for one, has remained “staunchly committed to the idea that meta-analyses must deal with all studies, good bad and indifferent” (para. 33). Restricting research syntheses to studies that meet methodological standards may be an effective tool for estab- lishing evidence-based practices, but choosing only well-designed studies does not inoculate research syntheses from the influence of constructs and the assumptions they carry.

Under previous positivist notions of social science inquiry, con- structs, thought of as latent variables, were operationally defined in order to sanitize them of tacit assumptions (Phillips & Burbules, 2000). Such treatment rested on a hard distinction between theoreti- cal constructs, which were loaded with assumptions, and empirical constructs, which were taken to be assumption free. But the distinc- tion between these two types of constructs failed to hold after Kuhn (1962) showed that scientific communities work within prevailing frameworks (paradigms), which are central to empirical claims. Despite advantages that may accrue from operationally defining constructs in the process of a research synthesis, doing so does not isolate constructs from their social, political, historical, and cultural contexts. The identification of underlying assumptions in the research synthesis process is not a warrant for “anything-goes” rela- tivism or a call to increase objectivity. Rather it supports the position that methodological decisions about what research designs to include in a research synthesis, and constructs that inform and emanate from those decisions, ought to be unpacked and subject to critique.

The recognition and use of critique in reviews is not new. Cooper (1985) noted early on that the purpose of conducting a review could be critical to show how previous conclusions were unwarranted based on “the literature’s incommensurability with the reviewers’ theoretical stance and/or criteria for methodological validity” (p. 10). Similarly, Petticrew and Roberts (2006) described critical reviews as aiming to critique methods and results of primary litera- ture but without “using the formalized approach of a systematic review” (p. 41).

A host of qualitative systematic review methods now employ more formalized approaches to critique primary literature in the review process. For example, the metasynthesis phase of metastudy explores how various theoretically informed analytic options influ- ence research findings (Paterson et al., 2001). Dixon-Woods and colleagues (2006) also characterize CIS as not just summarizing the literature’s data but also tracing the sociopolitical origins of entrenched constructs. However, CCS stands out from these sys- tematic review methods because it takes research synthesis as its object of critique and shows how research synthesis results are sensi- tive to constructs in the literature it includes.

Introducing CCS

Before demonstrating our use of CCS, we describe it in terms of its philosophy of inquiry, methodology, and methods.

Philosophies of inquiry are attempts to provide consistent answers to a constellation of questions, such as What is the notion of inquiry? What should count as evidence in educational inquiry? (Biesta, 2015). Methodology delineates what the inquiry entails and provides justifications for how it will be conducted within the philosophies of inquiry it is embedded. Methods describes the procedures scholars follow while inquiring within the overall methodology.

Philosophy of Inquiry

We conceptualize CCS as philosophically dexterous, transferable among different paradigms of inquiry within the critical tradi- tion. By critique, we mean a sociopolitical analysis that highlights how power and ideology operate to structure and stratify society, to marginalize, oppress, and limit possibility. Below we illustrate this dexterity with two stances: critical realism (Bhaskar, 1998) (informed by Marxism) and poststructuralism (e.g., Foucault, 1980). Although these stances diverge in many respects, they can be situated within the critical tradition and thus supply the intellectual resources needed for CCS.

Critical realism. Realism, including critical realism, holds that the deep structure of reality is external to human cognition (Gorski, 2013; House, 1991). Although reality may be multi- layered (Bhaskar, 1998), realism values constructs that map onto external reality (Sider, 2014). The construct “human being,” for example, may directly map onto reality’s structure (Armstrong, 1978), and if so, any full cognition of reality would require the inclusion of that construct to be adequate. It would not be a proper object of criticism. If “human being” does not map onto reality, however, then it can be the object of criticism and aban- doned, revised, or refined to meet social needs. Constructs such as “disability” may fit this latter category (Searle, 1997). Empiri- cal inquiry can determine if such constructs actually function to legitimate social oppression and so neutralize them. A dosage of criticality is thus injected into realism when advocates of realism recognize the need to unmask hegemonic constructs wrongly assumed to map onto reality, hence, critical realism (Bhaskar, 1988, 1993).

Following Ian Hacking (2002, 2004), critical realism pos- its a dynamic relationship between reality and dominant ide- ology, which creates a looping effect. For instance, when researchers classify a subset of schoolchildren as “high achiev- ers,” this dynamic relationship is reflected when schoolchil- dren accept, resist, or redefine this identity option. Their response can reinforce or discourage researchers from using the construct or adapting it. The “high achievers” construct thus evolves due to a feedback loop between labelers and labeled. Looping makes vocabulary in education different from vocabulary in natural science, wherein no such interac- tion between labeling and labeler exists (e.g., rocks do not contest or admire geologists’ labels of them). A critical real- ist–informed CCS might evaluate the impact of looping effects on research syntheses to provide emancipatory expla- nations of findings. For example, one might wonder if the suspect construct of “disability” harmfully delimited the pos- sible findings of a research synthesis.

134 EDUCATIONAl RESEARCHER

Poststructuralism. In contrast to critical realism, poststructural- ism holds that reality is entirely ideological. Poststructuralism engages boundaries between prevailing ideologies and sees real- ity as blurred (Peters & Burbules, 2000), one can never step outside of ideology to “objectively” define reality according to neutral facts. No external or objective structure (like the looping effect) exists to distinguish between what is real and what is defined by a construct. A poststructuralist analysis, for example, would critique the “looping effect” as producing a distinction between “labeler” and “labeled,” rather than reflect- ing a “real” process.

Drawing on Foucault’s (1980) description of power and knowledge (power/knowledge), a poststructuralist analysis for- mulates ideology as laden with power. The role of power in lever- aging some ideologies above others (e.g., neoliberalism) takes many forms and is often masked in liberal democracies (Giroux, 2004). But the incessant flux and productive work of power means no ideology ultimately prevails over the others or even remains stable over the long haul (Foucault, 1980). Ideological disagreements are not resolvable by evidence but through power. A poststructuralist-informed CCS might seek to disrupt prevail- ing ideologies in research syntheses by showing that the con- structs studied are not inevitable, natural, or immutable but instead contestable as they manifest through fallible social and historical discourse.

Methodology

CCS draws on two types of well-established methodologies in qualitative research. The first methodology is the qualitative sys- tematic review (Hannes & Macaitis, 2012), most closely resem- bling the processes of a CIS (Dixon-Woods et al., 2006). CIS involves an iterative and reflexive approach to synthesizing a body of literature on a topic (Dixon-Woods et al., 2006). Our CCS, although similar in approach, differs from a CIS in its aim. The aim of CCS is not to generate an interpretive synthesizing argument across the literature as is done in a CIS but to under- stand how the results of research syntheses are sensitive to the constructs included and excluded from review.

The second methodology is any “construct analysis” approach to analyze latent meaning in text. For example, constructs might be analyzed using techniques from ethnographic content analy- sis (Altheide & Schneider, 2013), semiotic analysis (e.g., Barthes, 1964), and/or (critical) discourse analysis (e.g., Gee, 2010). These analytic techniques assist researchers in interrogating con- structs and their assumptions in research syntheses and stem from the broad swath of the critical tradition.

Method

CCS involves multiple steps: (1) Identify a research synthesis of interest. (2) Focus on a bundle of constructs encompassed in the topic of interest (e.g., kinds of people, outcomes, interventions, or contexts). (3) Replicate the synthesis’ literature search strat- egy. (4) Screen gathered articles to eliminate works not on topic. The first four steps largely replicate the original synthesis, whereas the next steps are unique to CCS: (5) Divide the

remainder into distinct sets, literature included and excluded from the original synthesis. Importantly, the excluded set should include conceptual and empirical research originally deemed not up to par. (6) Develop coding sheets that track key characteris- tics of construct formulation (e.g., questions posed, key terms defined, answers given). (7) Code both sets of literature, included and excluded. (8) Resolve any coding discrepancies. (9) Proceed to analyze the constructs of interest in both sets of articles. Comparison of the two sets can identify what assumptions underlying the constructs informed the findings of the original research synthesis and how findings were sensitive to them. These steps define the method, but it need not be a linear pro- cess. Researchers may cycle back and forth between the steps, as needed.

Demonstrating CCS

Below we demonstrate how we proceeded through eight steps of CCS to explore the constructs “work” and “autism” in literature on employment for youth with autism (Wolgemuth et al., 2016) from a research synthesis on postschool transition for youth with disabilities (Cobb et al., 2013).

Step 1: Identify a Research Synthesis of Interest

The primary aim of CCS is to critically explore constructs in an existing research synthesis. Therefore, in the first step, the research team selects a research synthesis of interest. Reasons for selecting a particular synthesis will vary, but it is important that the search strategy used in the original research synthesis be pub- lished or otherwise known so that it can be replicated.

We chose to consider a research synthesis based on the first author’s participation in a U.S. Department of Education– contracted systematic review of literature on postsecondary out- comes for youth with disabilities (Cobb et al., 2013) that used WWC guidelines (with some modifications) to screen primary literature. The aim of the research synthesis was to identify effec- tive programs and strategies that support students with disabilities in the United States to transition from high school. Reflecting on the synthesis and its processes, the first author worried that of the 738 studies passing the abstract screening process, only 16 met WWC standards (with reservations). She wondered about assump- tions the constructs “disability,” “employment,” “independent liv- ing,” and “postsecondary education” advanced in the final report and what understandings might have been enabled in a more inclusive review. To explore these questions, she brought together a team of researchers to study a subset of literature in the research synthesis: literature on autism and employment.

Step 2: Focus on Constructs That Encompass a Topic of Interest

In Step 2, the review team determines the constructs it wants to explore, including its rationale for doing so. The first author noted that three of the 16 studies that met standards for inclu- sion in the research synthesis were about work for people with disabilities, but none was specifically about “work” for people

ApRIl 2017 135

with “autism,” although two included participants with autism diagnoses. She wondered what possible understandings of work and autism the overall report and its literature base privileged and excluded.

For our CCS, we focused on the constructs of “work” (con- ceptualized as both an intervention and an outcome construct in the research synthesis) and “autism.” We chose to make these constructs focal given a global increase in the prevalence of autism diagnoses (Elsabbagh et al., 2012) and reported lower employment and compensation of people with autism in the United States as compared to other disability categories (Wehman et al., 2014). We argued that although literature examined the construction of autism in fiction literature (e.g., Hacking, 2009), its construction in the academic literature had not been investigated.

Step 3: Replicate the Literature Search Strategy

In Step 3, the review team identifies the original search strategy; determines, based on the constructs it wishes to explore, whether terms need to be added and/or eliminated; and replicates the search using the original databases.

The research synthesis reported the key terms and Boolean operators used to search the literature: disability (e.g., autis* OR Asperger*), AND population (e.g., adolescent OR youth) AND outcome (e.g., work OR job OR employment) AND program (e.g., work experience OR supported employment) AND research design (e.g., RCT OR quasi-experimental). Because the original synthe- sis sought to synthesize literature for three postschool outcomes (employment, higher education, and independent living) for all youth with disabilities, we replicated the search for our CCS using only terms that would yield literature on work for youth with autism. We eliminated key terms in disability and outcome and program that were not associated with autism and employ- ment to generate an initial set of primary literature germane to our review. That is, we removed disability key terms, such as intellectual disability; program terms, such as independent living; and outcome terms, such as higher education.

Although the original synthesis had a broader scope with regard to disability, intervention, and outcome, the methodolo- gies it sought were only those with potential to meet WWC evi- dence standards (e.g., single-case designs, quasi-experimental designs, and randomized controlled trials). We therefore added key terms in research design, whose original set included only terms associated with quantitative studies, to capture conceptual and qualitative work (e.g., qualitative, commentary). We ran our literature search using the same databases (e.g., ERIC, PsycINFO, Medline) as the original review, which yielded 13,076 sources. Also duplicating the original review, we limited our search to only research published in peer-reviewed journals. We recognize this as a source of publication bias that is a limitation of the original review and a delimitation to the scope of our critique.

Step 4: Screen Gathered Articles to Eliminate Works Off Topic

In Step 4, the review team screens the articles identified in the search strategy following the original review parameters and, if

applicable, narrowing those parameters based on the constructs the team chose to explore.

We began screening the 13,076 articles by first identifying only those that contained the words autism or Autism Spectrum Disorder or ASD or Asperger’s Syndrome and work or employment or vocation or job or career either in their titles or abstracts. We chose to conduct this initial screen in order to more efficiently eliminate articles that were not of interest to our review and to yield a more manageable set of article abstracts to screen for a team of four researchers. This process left us with 2,738 articles. Consistent with the original review, we then screened the abstracts to identify articles that discussed work training or employment that did or could occur during high school. Also consistent, we defined (a) work as employment, vocation, or par- ticipation in paid or unpaid labor; (b) transition programs as including career training, career therapy, or counseling; and (c) youth as people between the ages of 13 and 22. We included lit- erature about adults when it offered retrospective examinations of their work experiences as youth. As is typical in this stage of the review process, we erred on the side of inclusion. The abstract screen process resulted in 252 sources for which we conducted a full-text screen. The full-text screen yielded 62 articles for extraction.

Step 5: Develop an Extraction Pro Forma

In Step 5, the review team develops a pro forma to extract key information from the articles and capture all statements in the articles about the constructs of interest. Dixon-Woods and col- leagues (2006) reported that the pro forma they developed for their CIS was ultimately impractical to use, especially for large documents, and wondered at the utility of formal data extraction for interpretive syntheses. Given the focused nature of CCS on specific constructs and the inclusion of only published articles in the original review, we felt a pro forma would help us identify features of articles and specific sections for later analysis.

Our pro forma, summarized in Table 2, included summary information about the article, including its purpose, methods, population studied, findings, and conclusions. It also included a section about methodology to enable us to analyze potential connections between methodology and constructions of autism and work. The remainder of the pro forma focused on the con- structs of interest (autism, work, and the worker with autism). We used the pro forma to record positive and negative state- ments/definitions of autism, work, and the worker with autism. By positive and negative, we did not mean “good” and “bad” but meant statements about what autism/work is (positive) and what autism/work is not (negative). We also included spaces for notes on our initial impressions of the constructs and other observations.

Step 6: Use the Pro Forma to Extract Information

In Step 6, the research team uses the pro forma to extract rele- vant information about the articles and statements about the constructs of interest. This process will likely be iterative with Step 5; that is, several team members can use the first draft of the pro forma to extract one study, come together and discuss their

136 EDUCATIONAl RESEARCHER

findings, identify commonalities, and potentially revise the pro forma to better suit their research aims. Step 6 is also likely to result in further elimination of articles based on the inclusion/ exclusion criteria in Step 4.

We used our pro forma to extract information from the 62 articles that passed the full-text screen. We extracted the first eight articles as a team and met twice to share our extractions, reconcile differences, and discuss the pro forma. These initial extractions resulted in some revisions to the pro forma (e.g., we clarified what we meant by positive and negative). Once we felt confident in our process, we extracted the remaining 54 articles in pairs. The pairs conducted their extractions individually and then met to reconcile differences and create a final pro forma. During this time, we met biweekly as a team to discuss our prog- ress and any questions or concerns. This process also resulted in the elimination of an additional 45 articles that did not meet the aims of the original review (e.g., employment studies conducted in postsecondary, instead of secondary, settings

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.