Identify a research problem appropriate for a mixed methods research study and write a paper describing how you would address that research problem.

Mixed Methods

Qualitative and quantitative methods can be combined in a variety of ways to form what is called mixed research methods. This is an attempt to draw from multiple epistemologies to frame and understand phenomena. As a result, researchers are supposed to increase the validity of their studies (through triangulation) to allow them to reach generalizations that will support the formation of theories to accurately describe the phenomena under investigation. While combinations of methods seem to move away from the methodology and get closer to practice as an approach to examining a research problem, there are many researchers who view the methodology as a separate and independent epistemological way of approaching research. This brings mixed methods into the realm of pragmatism (somewhere in between positivism and constructionism). Others, though, believe that paradigms are not to be viewed as distinct, but rather as overlapping, or with fluid boundaries where one gives rise to and supports the other, so combining them is quite an acceptable way of conducting research. In any case, following a mixed methods approach should not be considered as an attempt to alleviate the disadvantages of one of the methodologies you have seen so far with the advantages of the other, but rather, as a need for properly addressing the research problem under investigation.

The main issue with mixed methods is the sequence and extent of applying the quantitative and qualitative components of inquiry. For example, you might be interested in understanding the context of a real-life phenomenon like leadership in transient teams (teams that are formed for a specific task and then dissolved). If you are not aware of the characteristics of leadership that lead to success or failure of leaders in such teams (or there is no strong research behind it), you might decide to begin with a qualitative methodology that through in-depth interviews with select leaders of transient teams will reveal key characteristics. You will then follow this with a quantitative methodology whereby with the development of a widespread questionnaire, you will prove that the identified characteristics (or some of them) exist universally in transient teams. Alternatively, you might already know (for example, from past research or similar cases) the range of possible characteristics, so you might decide to start with a quantitative methodology to identify the specific leadership characteristics that affect transient team performance, and then follow with a qualitative methodology (for example, in-depth interviews) to identify the reasons behind the influence of the specific characteristics.

There are three possibilities of mixed methods designs. You can start with a quantitative approach (large sample) that will provide findings like certain characteristics (attitudes, traits, perceptions, etc.) of a population and follow up with a qualitative approach (specific small sample) for explanations to surface, and thus the method is called explanatory sequential mixed methods. On the other hand, there are exploratory sequential mixed methods where you can start with a qualitative approach (small sample) to explore the range and depth of a phenomenon and continue with qualitative (to a wider sample) to provide the “proof” and confirm what you found. Finally, you can have both methods working in parallel (convergent parallel mixed methods) and aim at calibrating each of them and comparing their results to converge (or diverge) for an interpretation, enhancing in this way the validity of the research findings. An example of the parallel application of methods is when they are combined in one instrument, like a questionnaire that has both closed-ended and open-ended questions.

Apart from the sequence of execution of the various methodologies, of significant importance is dominance. The uncomplicated way of viewing dominance is in terms of the time devoted to each methodology. This is a simplistic view as their importance in research might be different. For example, one might dedicate plenty of time and resources in in-depth interviews to confirm past research findings while spending little time in posting a questionnaire online and processing the results. While the interviewing part might seem necessary, the importance of the quantitative part is at a much higher level.

Along with the advantages of mixed methods research like the well-rounded investigation of a phenomenon and the insight they could provide for guiding the future choice of methods, there are criticisms that need to be considered before adopting such methodologies. Prominent among them are the contradictions of the underlined paradigms that need to coexist and provide support for the research, as well as the competencies that the investigators need to have as they need to cover the full spectrum of quantitative and qualitative methodologies. Additionally, mixed methods studies take significantly more time and more resources to complete, making them unsuitable when time and resources are of the essence. In closing, mixed methods might be ideal for comparing quantitative and qualitative perspectives for instrument calibration, for discovering the parameters and variables involved in phenomena, for understanding and providing support with raw/quantitative data for the “how” and “why” of phenomena, and in support of interventions and change initiatives (like for marginalized populations).

Develop a Mixed Methods Research Strategy

Instructions

For this assignment, you must identify a research problem appropriate for a mixed methods research study and write a paper describing how you would address that research problem.

Be sure your paper includes the following:

Problem statement

Purpose statement

Data collection plan

Data analysis plan

Length: 3 to 5-page academic paper, not including title and reference pages

References: Include a minimum of 3 scholarly resources.

The completed assignment should demonstrate thoughtful consideration of the ideas and concepts presented in the course by providing new thoughts and insights relating directly to this topic. The content should reflect scholarly writing and current APA standards.

Requirements: Length: 3 to 5-page academic paper, not including title and reference pages

Sage Research Methods Qualitative and Mixed Methods in Public Health For the most optimal reading experience we recommend using our website. A free-to-view version of this content is available by clicking on this link, which includes an easy-to-navigate-and-search-entry, and may also include videos, embedded datasets, downloadable datasets, interactive questions, audio content, and downloadable tables and resources. Author: Deborah K. Padgett Pub. Date: 2014 Product: Sage Research Methods DOI: https://doi.org/10.4135/9781483384511 Methods: Case study research, Mixed methods, Grounded theory Keywords: social capital, surveying Disciplines: Health, Nursing, Sociology, Medicine Access Date: September 21, 2023 Publishing Company: SAGE Publications, Inc. City: Thousand Oaks Online ISBN: 9781483384511 © 2014 SAGE Publications, Inc. All Rights Reserved.

Mixed Methods The “new era” of method integration (Tashakkori & Creswell, 2007, p. 3) can be seen as a pragmatic response on the part of researchers wanting to maximize their understanding of a particular problem (Johnson & On-wuegbuzie, 2004; Morgan, 2007). Some research topics (such as public opinions about childhood vaccines) are manifestly quantitative; others (such as IV drug users’ needle–sharing practices) are undoubtedly qualita-tive. In the vast middle ground lie many opportunities to use both approaches for synergistic ends. At the same time, mixed methods designs are complicated and sometimes messy affairs (Creswell & Plano Clark, 2010). Integrating the quantitative and qualitative “sides” poses epistemological and logistical chal-lenges that few research courses address (Tashakkori & Teddlie, 2010). That said, the momentum behind this trend is unlikely to slow down anytime soon. The Rise of Mixed Methods and Their Rationale(s) Just as the term qualitative methods came of age in the 1970s, mixed methods is a fairly recent addition to the research lexicon (Tashakkori & Teddlie, 2010). Formerly (and sometimes still) referred to as “multimethod,” “multistrategy,” or “triangulation by method,” mixed methods studies currently offer a wide and at times con-fusing array of options (Bryman, 2006; Creswell, 2007; Tashakkori & Teddlie, 2010). Ethnographers and others have a long history of including quantitative data and analyses. This unheralded “mixing” lost favor as the methods became more interview–based with the rise of grounded theory and narra-tive approaches. In addition, mixing on the qualitative side was derogated in critiques asserting the incompat-ibility of positivist assumptions of realism with constructivist assumptions of multiple interpretations (Lincoln & Guba, 1985; Morgan, 2007). On the quantitative side, the soaring dominance of quantification and statistics by the mid–20th century cast doubt on the value of qualitative data with its small samples and presumed lack of generalizability. Pioneering quantitative methodologists (D. T. Campbell & Stanley, 1963; Cook & Campbell, 1979) acknowledged the util-ity of qualitative data, but only in a supplementary or minor role. Interestingly, D. T. Campbell (1979) recanted his earlier criticisms of case studies and went further to state that conflicting results in mixed methods studies should cast the quantitative results as suspect “until the reasons for the discrepancy are well understood” (p. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 2 of 18Qualitative and Mixed Methods in Public Health

52). Caracelli and Greene (1997) discuss three reasons for carrying out mixed methods studies, which include triangulation, complementarity, and expansion. Triangulation, the earliest and most widely invoked of ratio-nales, refers to comparisons for purposes of corroboration (Morse, 1991). Because triangulation presumes a fixed point of reference waiting to be converged upon, its use has been criticized as naïve and sometimes misleading. It also raises questions when findings are divergent rather than convergent (Flick, 2004; Sande-lowski, 2000). Complementarity refers to enhancement or clarification. Thus, the quantitative and qualitative substudies represent different pieces of the puzzle. Expansion refers to presenting “side–by–side” or juxta-posed findings to keep them intact (Caracelli & Greene, 1997). Types of Mixed Methods Research All research designs operate from a premise of intentionality, and mixed methods designs point to the desire to link or integrate. As such, they portend specific procedures to carry this out (Haase & Myers, 1988). Inter-estingly, few mixed methods studies in the published literature use the terminology and notational systems promulgated by leaders in the field (Creswell, 2007; Morse, 1991; Tashakkori & Teddlie, 2010). Neverthe-less, it helps to recognize the many options and approach them in a systematic way. As shown in Table 3.1, the options in designing a mixed methods study involve two primary axes—sequential versus concurrent and dominant/subdominant versus equal (Creswell, 2003; Miller & Fredericks, 2006). Dominant refers to which method is given more weight and prominence in the study. When examining a study’s written report, this can be glaringly obvious or deeply ambiguous, depending on the study’s design and how clearly it is described. Sequential versus concurrent axes refers to the timing of the methods, whether used one at a time or simul-taneously. Table 3.1 Mixed Methods Designs Arranged by Timing and Dominance Sequential Concurrent DominantÐLess Dominant CELL 1 QUAL → quan qual → QUAN CELL 2—ÒNestedÓ QUAL + quan QUAN + qual Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 3 of 18Qualitative and Mixed Methods in Public Health

Using established notations of capital letters (for dominance or priority), arrows (for sequencing), and plus signs (for simultaneity; Morse, 1991), Table 3.1 shows various possibilities for the sequential and concurrent designs. As might be expected, QUAN–dominant designs are more common than QUAL–dominant designs and dominant/less–dominant designs are more common than equally weighted designs. Both of these obser-vations are a reflection of the way the research world is organized and the tendency to conserve resources and/or favor one method over another. A caveat before we go further: These design types are offered primar-ily as a heuristic device. In practice, mixed methods studies are complicated and not so easily categorized (Miller & Fredericks, 2006). Sequential Designs In sequential designs, how the study’s segments are prioritized and integrated depends on its priorities. As shown in Cells 1 and 4 of Table 3.1, this can occur in six different ways. Among the dominant/less–dominant designs, the most common (qual→QUAN and QUAN→qual) typically involve using focus groups or individual interviews to prepare for the “main event” (survey, instrument development, experimental trial, etc.) or to bet-ter understand it after the fact. The ecological validity of a quantitative study can be enhanced considerably by grounding the study in qualitative interviews and observation before or after. Conducting focus groups with students and teachers before implementing a safe–sex education program is one example of this approach; positioning the focus groups afterward is another example, albeit for a different purpose. QUAL→ quan and quan→ QUAL studies position their quantitative segments as less dominant. An example of the first would be an intensive qualitative case study of an innovative program for individuals with multiple sclerosis that is used to develop questions for a brief online survey of agencies serving MS patients. In the reverse sequence (quan→ QUAL), a telephone survey of parents in a school district might be used to select a QUAN → qual quan → QUAL Equal Weighting CELL 4 QUAL → QUAN QUAN → QUAL CELL 3 —ÒFully IntegratedÓ QUAL+QUAN Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 4 of 18Qualitative and Mixed Methods in Public Health

subsample willing to be interviewed in depth about their experiences with the district’s new program on child-hood obesity. Although less common, equal weighting in sequential designs (Cell 4 of Table 3.1) means that both qualitative and quantitative segments receive sufficient allocations of resources to meet their respective sampling and data quality needs. Concurrent Designs In concurrent designs, one method may be dominant over the other (QUAN+qual or QUAL+quan) or they may be given equal weight (QUAN+QUAL). As mentioned earlier, “dominant/less dominant” or nested designs (Cell 2 of Table 3.1) are much more common (Creswell, 2007). Box 3.1 offers an example of a QUAN+qual study carried out in different nations. Box 3.1 A Mixed Methods Study (QUAN+qual) Testing a Measure of Social Capital in Peru and Vietnam Mixed methods have an intrinsic appeal for instrument development and testing because most measures’ un-derlying constructs are complex and open to differing meanings and interpretations. One such concept, that of social capital, has become widely used as an indication of the ways that social relationships may confer health benefits, from fostering a sense of belonging to providing links to valuable resources. The measure-ment of social capital at the individual level is seen as a potential indicator of health in general and access to health care in particular (Szreter & Woolcock, 2004). DeSilva and colleagues (2006) developed a measure of social capital (the SASCAT), translated it into Spanish and Vietnamese, and administered it to a large sample of children’s caregivers (3,000 in Peru and 2,771 in Vietnam). In addition to psychometric tests of the mea-sure’s validity, the researchers criterion–sampled 20 Peruvian and 24 Vietnamese respondents for in–depth interviews. These “cognitive interviews,” lasting from 1 to 2 hours, elicited further thoughts and ideas related to each SASCAT item. An example of an item is, “In the past 12 months, have you joined together with other community members to address a common problem or issue?” The interviews were audiotaped and content analyzed to see if (and how often) open–ended comments diverged from the authors’ original intention re-garding each item’s meaning. The findings were revealing. Although the quantitative factor analysis results from the two countries were strikingly similar, the qualitative interviews brought several cultural misunder-Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 5 of 18Qualitative and Mixed Methods in Public Health

standings to the surface. The concept of “community,” for example, was readily accepted in Vietnam but not understood by many Peruvians (who defined it as one’s social support network, not the surrounding area). In both Peru and Vietnam, “trust” was not considered something one can impute to the “community” in general but only to known individuals. Similarly, “help from others” was largely defined as economic support—contrary to the measure’s inclusion of emotional support within the definition. The authors understandably concluded that cognitive validation needs to precede instrument development. Commentary: This study’s QUAN+qual design was an ambitious and successful application of mixed meth-ods. The two methods were used for corroboration as well as completeness (i.e., the researchers did not posit a single meaning for each item but instead sought out multiple meanings to improve the measure). The find-ings demonstrate the critical importance of qualitative methods in cross–cultural research in which subjective meanings can vary along cultural lines. If this is not taken into consideration, quantitative data collection will be prone to error and misunderstandings. In QUAN+qual designs, researchers typically collect qualitative data to enliven or illustrate their quantitative findings, for example, excerpts from responses to open–ended questions or case vignettes (Morgan, 1997). In the reverse QUAL+quan approach, qualitative researchers might collect some quantitative data via stan-dardized measures or they might use supplementary quantitative data from documents or archives. Snow and Anderson (1991) made use of tracking data from various agencies to supplement their intensive inter-views and ethnographic observation of the homeless. The resulting depictions contained both statistical and ethnographic descriptions of their lives. Using quantitative data can be risky with small samples, but if done judiciously it need not detract from the inductive, emergent nature of a qualitative study. Similarly, the inclu-sion of ancillary qualitative data does not challenge the primacy of a “big QUAN” study. A QUAL+QUAN study (Cell 3) is among the rarest of mixed methods types due to aforementioned demands on time and resources as well as the challenges of fully integrating the two “sides.” Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 6 of 18Qualitative and Mixed Methods in Public Health

Mixed Methods: Ways of Going About It Structural and Design Decisions: What, When, Where, and How? Leaving the abstract realm of design types for real–world decisions about mixing methods requires that we unravel the research process and decide which phases will (or should) intersect and which will remain intact. Are there constraints on doing this, or can one mix and match at will? Consider the following series of state-ments: • Paradigms (post–positivist, constructivist, critical) do not dictate methods (grounded theory, phenom-enological, experimental/quantitative, surveys, etc.). • Methods do not dictate data collection techniques (interviews, questionnaires, observation). • Techniques of data collection do not dictate data analyses. Such assertions are strongly opposed by postmodern contentions that one cannot mix positivist and construc-tivist epistemologies (Lincoln & Guba, 2000). But such objections have not slowed the movement toward mix-ing below the paradigmatic level (Morgan, 2007). (The reader might want to return to Figure 1.1 in Chapter 1 on page 15, which shows the downward line or spiral of a study.) Thus, a grounded theory study can be car-ried out using post–positivist or constructivist epistemologies; some studies appear to do both simultaneously. At a lower level, many a study has transformed qualitative data into numbers. To be sure, some combinations do not work, for example, narrative analysis and quantitative data. Moreover, one should not mix and match willy–nilly without considerations of fit and appropriateness. According to Sandelowski (2000), most mixing takes place “on the shop floor of research” (p. 246) during sampling, data collection, and data analysis. Tashakkori and Creswell (2007) discuss dual dimensions to sam-pling (probability and purposive), data collection (quantitative and qualitative), data analysis (statistical and thematic), and presentation of the findings (numeric and narrative). The points of contact between the quantitative and qualitative sides can be many or few. Sequential designs leave open the opportunity for each substudy to remain intact (assuming a reasonable connection is made). In concurrent designs, the parallel processes, or “strands” (Tashakkori & Creswell, 2007, p. 3), may intersect at one or more phases. The lowest level of mixing intensity is when the two sides stay separate and come together only at the end when findings are juxtaposed. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 7 of 18Qualitative and Mixed Methods in Public Health

Mixing at the data analysis level, according to Tashakkori and Teddlie (2010), may include “qualitizing” quan-titative data and its opposite process of “quantitizing” qualitative data. The latter of these, which refers to converting qualitative data into numbers or variables, has a long history in content analysis. (An example of “quantitizing” is provided in Box 3.2.) Sandelowski (2000) “qualitized” her quantitative data by creating profiles or categorical types from scores on standardized measures. Box 3.2 ÒQuantitizingÓ Data in a Grounded Theory Study of Breast and Prostate Cancer Online Discussion Boards Online chat rooms and discussion boards offer an abundance of narrative data for qualitative analysis. Good-en and Winefield (2007) used grounded theory and a “quasi–numeric” (p. 103) approach to examine gen-der differences in language styles and communication among cancer survivors communicating online. They started with a hypothesis positing greater use of emotional communications by women and greater use of informational communications by men. They examined online communications among 69 women with breast cancer and 77 men with prostate cancer by using open, axial, and selective coding conducted independent-ly by two readers. The number of codes per message (or posting) and the frequency with which individuals posted were calculated and displayed in tables in the published article. From these analyses, two selective codes (“information support” and “emotional support”) were identified along with their respective axial and open codes. Examples of axial codes included “facts about the disease” (under information support) and “cop-ing philosophies” (under emotional support). Instances of open codes were counted in each database and categorized proportionately under each of these two main headings. As a result, Gooden and Winefield found that information communication comprised 60% of women’s communications and 64% of men’s communica-tions. Thus, there were modest (and probably non–significant) gender differences in the frequency of emo-tional (versus informational) communication. Virtually all of the results section of the article was devoted to describing the codes, thereby revealing subtle but meaningful aspects of gender. Under “information support,” for example, men were likely to offer detailed factual information compared to briefer informational summaries supplied by women. Under “emotional support,” women used warm dialogue and affectionate phrasing, while men suggested to their peers that they “keep their chin up” and “beat the bastard.” In other code domains, such as use of humor and group spirit, men and women did not differ. Commentary: This study’s use of a hypothesis and a QUAL+quan design set the stage for the “quantitizing” that followed. However, the quantitative findings comparing men and women were modest and anticlimactic. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 8 of 18Qualitative and Mixed Methods in Public Health

In the study’s write–up, the numbers told a small story, but the qualitative themes and interpretations were the main event. Box 3.3 The Difficulty in Ascertaining Research Designs in Mixed Methods: An Example from a Study of Rural Bangladeshi Couples and Pregnancy Termination Few published reports of mixed methods studies use the typologies and design notations described in this chapter. Journal reviewers are not likely to demand them, and the complexities of mixing methods do not al-ways map onto extant typologies. A mixed methods study published in the American Journal of PublicHealth in 2008 offers a case in point. In the article, the authors Gipson and Hindin report using mixed methods to understand how rural Bangladeshi couples make family planning decisions including pregnancy termination. To do so, they draw upon health survey and surveillance data from 3,052 couples as well as 84 in–depth interviews conducted with 19 couples. Quantitative survey questions about childbearing and pregnancy ter-mination were a key interest in the qualitative interviews, but the investigators used a life history technique in the latter to avoid appearing intrusive or invading of the couples’ privacy. Although the authors do not say, the design appears to be sequential in that the quantitative data came from surveys (conducted from 1998 to 2003), and the qualitative purposive sample was drawn from a roster of couples who were enrolled in the survey as of 2004. On the other hand, the study design could be seen as concurrent since it juxtaposes the two “sides” without reference to the time lapse rather than presents them as temporally separate. It is also difficult to decide whether the design is dominant or equally weighted. It appears to be QUAN–dom-inant, since the findings section gives full coverage to the statistical analyses in tabular and text format fol-lowed by reference to the qualitative findings with selected illustrative quotes. The size and volume of the quantitative data, combined with a foreshortened qualitative data analysis (described by Gipson and Hindin [2008] as “focused” [p. 1828]), reinforces this notion of QUAN–dominance. However, this scenario is not en-tirely borne out in the weight the authors give to the two sides while discussing the findings and their impli-cations. For example, the qualitative results offer insightful perspectives on a number of topics including the stigma of having a child when the older children are nearing marital age (thereby hurting the older children’s Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 9 of 18Qualitative and Mixed Methods in Public Health

marital prospects), the use of traditional forms of abortifacients such as roots and homeopathic tablets, and the hidden ways that women terminate pregnancy without telling their husbands. All of these were the result of multiple in–depth interviews conducted with husbands and wives separately. It is hard to imagine how sur-veys could have brought forth such deeply sensitive information. With regard to the “what” question, Bryman (2006) reports that the concurrent mixing of standardized surveys and qualitative interviews is most common, the latter often based on a purposively selected subsample from the larger survey sample. From the qualitative side, focus groups are a popular choice for mixing; life history interviews and ethnographic observation are less amenable to mixing. From the quantitative side, random-ized clinical trials offer less fit for mixed methods compared to standardized interviews and surveys. Some Examples of Mixed Methods Studies The following are a few iterations of mixed methods designs with hypothetical examples. • A sequential design in which scores on an instrument administered during a survey are subsequently used for criterion sampling of a small subsample for qualitative interviews. For example, a study of depression in college students might use scores on the depressive symptom scale to identify stu-dents at highest and lowest risk. These students could then be interviewed in depth about their col-lege experiences and life stressors. • A concurrent design at the data collection stage in which inÐdepth interviews are paired with Lik-ertÐtype survey questions. For example, a study of South African women might administer measures of exposure to partner violence in a community meeting along with post–meeting focus groups for volunteers willing to discuss the issue at greater length. • A concurrent design at the data analysis stage in which qualitative data are ÒquantitizedÓ and con-verted to categorical variables and tested using statistical analysis. In the NYSS, for example, we conducted case study analyses and categorized participants by whether they were “users” vs. “non–users” of drugs and alcohol during their year–long participation in the study. This dichotomous yes/no variable was the outcome in a series of logistic regression analyses comparing individuals grouped by race, housing status, gender, and so on (Padgett, Stanhope, Henwood, & Stefancic, 2011). Although our sample size was fairly large for a qualitative study (N = 75) and enabled use of Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 10 of 18Qualitative and Mixed Methods in Public Health

multivariate analyses, smaller–scale studies might use bivariate analyses or non–parametric statis-tics such as Fisher’s exact test. • A sequential design in which quantitative data are ÒqualitizedÓ (i.e., statistical analyses are used to produce profiles or clusters that set the stage for qualitative interviews with study participants with-in each cluster to corroborate these analyses). A standardized interview of gay male adolescents might, for example, be used to create a typology of HIV risk based on the scores on measures of substance abuse, depression, and social support. Respondents who fit the “high risk” profile could be interviewed in depth and contrasted with those who fit the “low risk.” • A longitudinal concurrent design in which the quantitative and qualitative sides mutually inform one another. An example is shown in Figure 3.1 for a hypothetical community–based intervention. As shown, focus groups with key stakeholders in the community help guide the content for a household needs assessment survey. Findings from the survey help shape the intervention (e.g., a nutrition pro-gram for low–income parents) that is pilot–tested and then implemented. Throughout the interven-tion, ethnographic observation is used to help evaluate process and fidelity aspects of the implemen-tation, identify problem areas, and collect data from participants (children, parents, school officials) assessing effectiveness. Depending on the study’s aims, the ethnographic data may be confined to informing the quantitative work, or it may be separately presented as findings and compared (or tri-angulated) with the quantitative outcome data. • A longitudinal sequential design oscillating between the quantitative and qualitative sides. In contrast to the previous example, this design involves alternating between sides over time. As depicted in Figure 3.2, such oscillation could take the form of ethnographic observations on an American Indian reservation that are used to inform a community survey on needs of children and families. The sur-vey’s findings then lead to criterion–sampled focus groups of adolescents, the results of which inform a targeted intervention to prevent adolescent suicide. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 11 of 18Qualitative and Mixed Methods in Public Health

Figure 3.1 A Longitudinal Mixed Methods Design Figure 3.2 An Oscillating Mixed Methods Design Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 12 of 18Qualitative and Mixed Methods in Public Health

Writing the Mixed Methods Report As illustrated in the boxed inserts in this chapter, findings from mixed methods studies can be presented in a number of ways. Many published studies do not adhere to the terminology of mixed methods, thus making it difficult to ascertain what design they used (see Box 3.3 for an example). Assuming the two sides were kept intact, the findings are presented side by side, numerically, and thematical-ly. Usually some attempt is made to interpret them in tandem and discuss the degree to which they converge or diverge. Gioia (2004), for example, provided a descriptive summary of her quantitative findings along with a presentation of the qualitative themes, and then ended with a graphic display showing how the two “sides” were related. If the mixing occurred earlier such that the data analyses followed one approach, the findings are usually presented in one format. Perhaps not surprisingly given the motivation to mix methods in the first place, researchers tend to favor the multiple findings option to give each side its due. Challenges for Mixed Methods Studies Carrying out mixed methods studies poses challenges of various and sundry kinds. Among the logistical hurdles, most researchers are trained in one or the other method (most often quantitative), but not both. Having dual competencies in a team effort can overcome this drawback, but the lone investigator is at a disadvantage. Mixed methods studies also require dual outlays of time and resources to ensure that both “sides”—quantitative and qualitative—are given sufficient attention to be rigorous (Stange, Miller, Crabtree, O’Connor, & Zyzanski, 1994). Another logistical challenge accompanies oversight of the two sides when their rhythms and phases unfold in different ways. Qualitative data analyses, for example, start early in data collection and may result in going back out into the field for further sampling and data collection. Meanwhile, the quantitative side is proceed-ing in linear fashion, waiting out the data collection before beginning statistical analyses. The qualitative side, even though working with a smaller sample, takes considerable amounts of time and resources for transcrip-tion and data analysis not matched on the quantitative side. Given these tensions, the temptation to use a dominant–less dominant design is strong, and the quantitative side often comes out ahead in these situations. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 13 of 18Qualitative and Mixed Methods in Public Health

Because mixed methods imply combining within the confines of a single study, questions can arise regarding when and where one study ends and another begins. In sequential designs, too much time elapsed may lead to the conclusion that two separate studies were conducted. In concurrent designs, the “sides” may have few or no interactions and integration is minimal, thus giving the appearance of two unrelated side–by–side stud-ies. There are also questions about the adequacy and provenance of the data. Are answers to a few open–ended questions on a questionnaire sufficient to be considered qualitative data? Will a scale or index administered during in–depth interviews yield meager descriptive statistics of limited value? Collecting qualitative data un-der heavily quantitative auspices raises serious doubts about its authenticity and richness (Morse, 2005). Moreover, qualitizing quantitative data is not the mirror image of quantitizing qualitative data (this sentence is an admitted mouthful). With the former, quantitative data are aggregated and clustered, and the resulting categories are based on decontextualized data that are hardly comparable to categories inductively derived from in–depth interviews or extensive observation. Similarly, a disservice is often done to deep and rich data when they are quantitized. Perhaps the best lesson to come from all this is that qualitizing and quantitizing should be done with great caution and transparency regarding the reasons for doing so. The conundrum posed by triangulation also affects many a mixed methods study. When results from both sides are in accord, the researcher concludes (perhaps prematurely) that her findings are confirmed. As dis-cussed in greater detail in Chapter 9, the meaning of triangulation has been expanded beyond corrobora-tion to include completeness. Ethnographers, for example, often use quantitative and qualitative analyses for comprehensiveness rather than for validation. The problem arises over what to do when the qualitative and quantitative findings are neither convergent nor complementary. Some researchers present the two sets of findings, acknowledge the conflict, and ask the reader (and future researchers) to resolve the differences. Others use the discrepancy as an opportunity to inquire further, first to ensure that each of the “sides” is not flawed or biased in some way and then to examine and even use the discrepancy as an opportunity to broad-en or revise the study. Box 3.4 gives an example of a mixed methods study in which serendipitous findings from the qualitative “side” broadened the study and extended its impact to include new perspectives on barriers to breast cancer screen-ing. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 14 of 18Qualitative and Mixed Methods in Public Health

Box 3.4 Serendipitous Findings and Mixed Methods: An Example from the Harlem Mammogram Study The Harlem Mammogram Study was a mixed method (QUAN+qual) study examining why African American women delay in responding to an abnormal mammogram (Kerner et al., 2003). Interviews included measures of health locus of control, fear of cancer, beliefs about racism, insurance status, and psychological distress. The less–dominant qualitative portion consisted of questions about their mammogram experiences and dis-cussions of aging, racism, body image, and female sexuality. Data analyses were carried out separately by the quantitative and qualitative members of the team who had the requisite expertise. Both “sides” shared an interest in understanding delay (or timeliness) and maintained close contact as the analyses proceeded. The quantitative side used a multivariate logistic regression model to predict time to diagnostic resolution (within 3 months or longer). The qualitative side used a modified grounded theory approach to identify codes and themes. The qualitative data were intended to bring greater depth and understanding (a supplementary role), thus corresponding to “completeness” as a goal of triangulation. As it happened, they also contributed two serendipitous findings that would not have emerged otherwise. The first of these was the “air theory” of cancer which was subsequently noted in the literature (Freeman, 2004). This folk belief, which holds that opening the body surgically exposes it to air that can cause dormant cancer cells to grow and spread, was volunteered by several women during the qualitative interviews. Its relevance to our interest in delay was obvious because a surgical biopsy posed just such a threat. The second unexpected qualitative finding was about the physical and psychological toll of repeated diagnostic tests among women who had had multiple abnormal mammo-grams. A significant minority of women in the study had to undergo repeated and painful needle biopsies and other procedures (Padgett, Yedidia, Kerner, & Mandelblatt, 2001). The qualitative results were significant in ways that our multivariate model did not (and probably could not) take into account. How, after all, could we have anticipated or measured “air theory”? The results also turned out to have more grab than the statisti-cal analyses that proved to be disappointingly thin. Contrary to our hypothesized expectations, the effects of income, insurance coverage, and systemic barriers were not found to be statistically significant. Among 30 predictor variables tested in the multivariate model, only the degree of mammogram abnormality and whether the patient was given information were significant (Kerner et al., 2003). Had we omitted the qualitative portion of the study, the underperforming quantitative model would have been the study’s only finding for dissemina-tion. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 15 of 18Qualitative and Mixed Methods in Public Health

Mixed Methods Approaches to Program and Practice Eval-uation There are several ways that qualitative research can contribute to a mixed methods evaluation (Greene & Caracelli, 1997; Padgett, 2009b; Rallis & Rossman, 2003). Quantitative evaluations are good at establishing what works, but qualitative evaluations help to understand how a program succeeds or fails. Although admit-tedly more time–consuming, qualitative methods are less intrusive and less demanding than an experimental trial (Perreault, Pawliuk, Veilleux, & Rousseau, 2006). Qualitative researchers can fade into the woodwork and respond nimbly to the ebb and flow of organizational life. The addition of qualitative methods to a quanti-tative evaluation adds flexibility and depth (Drake et al., 1993). Qualitative research is particularly well suited to formative and process evaluation. Hong and colleagues (2005) used ethnography to conduct a formative evaluation of an HIV prevention program with injection drug users. Their findings regarding miscommunications and cultural relevance were used to inform and improve the intervention that resulted. Program or treatment fidelity studies are also amenable to qualitative inquiry. The inner workings of many programs—the dynamic interplay of the actors, their differing perceptions of events, and the effects of culture and gender—are difficult to anticipate and measure. Indeed, there is no substitute for what can be learned from extended ethnographic observation of a program and its day–to–day operations (Felton, 2005). Qualitative approaches also mesh well with social advocacy values in evaluation—they empower less power-ful stakeholders (clients, lower–level staff, etc.) by giving their voices greater prominence in “finding the value” of the program. A mixed methods approach cannot guarantee a successful evaluation, but it is likely to en-hance the depth and relevance of the findings. Summary and Concluding Thoughts The popularity of mixed methods is higher than ever before. Mixed methods approaches can bring unprece-dented synergy, but they are not a panacea. A number of methodological and logistical challenges stand in the way of successfully integrating quantitative and qualitative approaches, not the least of which are the ad-Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 16 of 18Qualitative and Mixed Methods in Public Health

ditional outlays of expertise, time, and resources needed to do justice to both “sides.” This chapter began with the archetypal possibilities based on sequencing and dominance. It also offered sev-eral examples of the complicated, even messy, mixed methods designs that characterize applications in re-al–world settings. The devil, as they say, is in the details. Determining what to mix from the qualitative and quantitative sides, when to mix them, and how to make the linkages takes careful advance planning. Not withstanding these challenges, mixed methods open the door to illuminating contrasts, whether done to cor-roborate, complement, or expand knowledge into new frontiers. Exercises 1. The Journal of Mixed Method Research debuted in 2007 as a reflection of the burgeoning interest in method integration. Browse this journal (or another of your choice) and locate a mixed methods study of interest to you. Using the notational system from Table 3.1, how would you characterize the study’s design? 2. In class discussion groups, take a look at Table 3.1. Which designs are most common and which are least likely to be used? Why? 3. Choose a public health program or practice of interest to you or your class discussion group and talk through how it could be evaluated using mixed methods. Choose a design from Table 3.1 and then specify how you would carry out the proposed evaluation. What types of quantitative and quali-tative methods, techniques, or analyses would you use? 4. Think of a topic for which mixed methods would not be a good fit. Why is this so? Additional Readings Creswell, J. W., & Plano Clark, V. (2010). Designing and conducting mixed methods research (2nd ed. ). Thou-sand Oaks, CA: Sage. Greene, J. C., & Caracelli, V. J. (Eds.). (1997). Advances in mixedÐmethod evaluation: The challenges and benefits of integrating diverse paradigms. San Francisco: Jossey–Bass. Johnson, P. J., & Onwuegbuzie, J. A. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 17 of 18Qualitative and Mixed Methods in Public Health

Miller, S. I., & Fredericks, M. (2006). Mixed–methods and evaluation research: Trends and issues. Qualitative Health Research, 16, 567–579. Sandelowski, M. (2000). Combining qualitative and quantitative sampling, data collection, and analysis tech-niques in mixed methods studies. Research in Nursing & Health, 23, 246–255. Tashakkori, A., & Teddlie, C. (Eds.). (2010). The SAGE handbook of mixed methods in social and behavioral research (2nd ed. ). Thousand Oaks, CA: Sage. Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781483384511 Sage© 2012 by SAGE Publications, Inc.Sage Research MethodsPage 18 of 18Qualitative and Mixed Methods in Public Health

OriginalresearchCriticalthinkinginstructionandtechnologyenhancedlearningfromthestudentperspective:AmixedmethodsresearchstudyRuthSwartUniversityofCalgary,FacultyofNursing,2500,UniversityDriveNW,Calgary,ABT2N1N4,CanadaarticleinfoArticlehistory:Received29December2015Receivedinrevisedform5September2016Accepted2February2017Keywords:CriticalthinkingTechnologyenhancedlearningenvironments(TELEs)OnlinediscussionforumsClassroomresponsetechnologyUndergraduatenursingstudentsInstructionalapproachMixedmethodsabstractCriticalthinkingisacclaimedasavaluableassetforgraduatesfromhighereducationprograms.Tech-nologyhasadvancedinquantityandquality;recognizedasarequirementof21stcenturylearners.Amixedmethodsresearchstudywasundertaken,examiningundergraduatenursingstudentengagementwithcriticalthinkinginstruction,platformedontwotechnology-enhancedlearningenvironments:aclassroomresponsesystemface-to-facein-classandanonlinediscussionforumout-of-class.TheCommunityofInquiryframedthestudycapturingconstructivistcollaborativeinquirytosupportlearning,andfacilitatecriticalthinkingcapability.Inclusionofquantitativeandqualitativedatasourcesaimedtogatheracomprehensiveunderstandingofstudents’developmentofcriticalthinkingandengagementwithtechnology-enhancedlearning.Thefindingsfromthestudents’perspectiveswerepositivetowardtheinclusionoftechnology-enhancedlearning,anduseinsupportingtheirdevelopmentofcriticalthinking.Studentsconsideredtheuseoftwoformsoftechnologybeneficialinmeetingdifferentneedsandpreferences,offeringvariedmeanstoactivelyparticipateinlearning.Theyvaluedcriticalthinkinginstructionbeingintentionallyalignedwithsubject-specificcontentfacilitatingunder-standing,application,andrelevanceofcoursematerial.Whilethefindingsarelimitedtostudentpar-ticipants,theinstructionalstrategiesandtechnology-enhancedlearningidentifiedasbeneficialcaninformcoursedesignforthedevelopmentofcriticalthinking.©2017ElsevierLtd.Allrightsreserved.1.IntroductionCriticalthinking,andexperiencewithtechnologyhavebeennotedasimportantqualitiesforgraduatestransitioningintopro-fessionalroles(BorglinandFagerstrom,2012;Floresetal.,2012;Lai,2012;MorrallandGoodman,2012;Roschelleetal.,2011;Saadeetal.,2012;Thomas,2011).Inhealthcare,criticalthinkingisessential,generatedbyconcernforpatientsafety,andcorrelatedwithimprovedpatientcare,developingnewknowledge,andqualitydecision-makingyieldingpositiveoutcomes(Burrell,2014).Practicingnursesusecriticalthinkingtoensurecompetent,appropriatecareintime-constrained,fluctuating,complexpatientsituations(Berkowetal.,2011;Feroetal.,2009).Withtheextentandeasyaccessibilityofinformation,nursingstudentsneedtobeabletogatheranduseitcompetently,purposefully,andefficiently;consequently,entailingtheneedforinquiry,criticalthinking,andreflectiontobeincludedinnursingeducationtodevelopdecision-makingcriticaltotheprovisionofsafe,qualitycareamidstcomplexandintensepatientsituations(Benjaminetal.,2013;Chan,2013;Romeo,2013).2.BackgroundResearchersandeducatorshaveemphasizedtheimportanceofdevelopingcriticalthinkinginnursingwhileacknowledgingvari-ationsinteachingstrategies(Chan,2013;Hunteretal.,2014;Krupatetal.,2011).Determiningwhatapproachtotakeinteach-ingcriticalthinkingandwhatinstructionalmethodtoapplyre-mainsachallenge.Behar-HorensteinandNiu(2011)reviewed42studiesoncriticalthinkinginstructionidentifyingmixedresultsfromsimilarinterventions.Someeducationalresearchershaverecommendedthatcriticalthinkingbemadeexplicittolearners,ensuringclarityaboutwhattheyareaimingtoachieve(Bensley,2011;Heijltjesetal.,2014;Kuetal.,2014;MarinandHalpern,2011;Yangetal.,2013).Thislendssupportfortheadditionofexplicitcriticalthinkinginstructiontobepurposefullyincorporatedintothenursingcourseofthisresearchstudy.E-mailaddress:rswart@ucalgary.ca.ContentslistsavailableatScienceDirectNurseEducationinPracticejournalhomepage:www.elsevier.com/neprhttp://dx.doi.org/10.1016/j.nepr.2017.02.0031471-5953/©2017ElsevierLtd.Allrightsreserved.NurseEducationinPractice23(2017)30e39

Theprevalenceoftechnologyentailstheneedforitsintegrationintoeducation(Kimetal.,2013;Lavinetal.,2010;OhandReeves,2014),leadingtothepotentialfortechnology-enhancedlearningenvironments(TELEs)insupportingstudentlearning.Brown(2013)depictsTELEsascomplexenvironmentswhichappropri-atelyusetechnologytosupportandimprovethequalityofthelearningexperience.Mergingface-to-faceandonlineTELEsun-dertakesablendedapproach,withpotentialtoengagestudentsactivelyinclass,andextendlearningbeyondthetimeandspaceconstraintsoftheclassroom,allowingstudentsthebenefitsofbotheducationalexperiences(Garrison,2013;GarrisonandVaughan,2008;Spiliotopoulos,2011).Variousauthorsagreethatstudentscanbeengagedindeeperthinkingwhencombiningdirectinter-actionandonlinecollaborativelearning(Akyoletal.,2009;Dziubanetal.,2011;KorkmazandKarakus,2009;Snodgrass,2011;Tappenden,2011;Yeh,2012).Thiswouldbebeneficialintheundergraduateclassroomswithlargerenrollmentswhereinvolvingstudentsindiscussionandreflectiontodevelopcriticalthinkingcanbeachallenge.Classroomresponsesystems,suchasTopHat,usedasin-classTELEshavethepotentialtoincreaseinteractivity,enhancepartici-pation,promoteengagementinactivelearning,fostercommuni-cationforclarificationandfeedback,andstimulatecriticalthinking,allofwhicharenotedofbenefitinhigh-enrollmentclassrooms(Bradyetal.,2013;DeBourgh,2008;MollbornandHoekstra,2010;Siauetal.,2006;TreesandJackson,2007).Thetypequestioningusedhasbeenhighlightedassignificanttofacilitateimprovementinstudentself-monitoringofunderstandingclassmaterial,andtopromotemetacognitiveabilitywhichisvaluabletostudentlearningandcriticalthinking(Anthis,2011).Severalauthorsrecommendtheneedtoconstructthelearningenvironmenttoencouragecomprehensionandapplicationoflearning;touserelevantcontentandappropriatetypesofquestionsandtogenerateenlighteningdiscussiontoengagestudentsandfostercriticalthinking(Beattyetal.,2006;DeBourgh,2008;MollbornandHoekstra,2010;ShiehandChang,2013).Theonlinediscussionforumastheout-of-classTELEoffersstu-dentsextensionoftheirlearningawayfromthebusyclassroomenvironment.Writtentextsemployedtobuildinteraction,communication,andshareperspectivesonlinecanovercometimeandspacelimitations(Dennen,2013;Garrison,2011).Dennen(2013)notestheasynchronousnatureofonlineforumssupport-ingcriticalthinkingthroughfosteringdepthandreflectioninstu-dents.Theuseofonlinediscussionforumscanprovidestudentswith“anytimeandanywhere”teachingandlearning,capacitytostorewrittendialogue,timetoprocesslearningandsharingofre-flectedthoughts,andcounterin-personshynesswiththeonline“unseen”forum(Arend,2009;CheongandCheung,2008;Pisutova-GerberandMalovicova,2009;Shieh,2010).Similarwiththeuseoftheclassroomresponsesystem,thepotentialofonlinediscussionsinfosteringcriticalthinkingisinfluencedbythequalityofthefacilitationandquestionsgenerated(RichardsonandIce,2010;SzaboandSchwartz,2011;Thormannetal.,2013).Thedistinctuseofclassroomresponsesystemsandonlinedis-cussionforumshavedemonstratedcapacityinengagingstudentsincriticalthinking.ThisresearchstudyundertooktoexamineintegratingthesetwoTELEsusingablendedapproach,anticipatingpotentialtofurtherengagestudentsincriticalthinkingbyprovidingalternativeenvironmentswhichcomplementandsup-plementlearning.SignificanttobothTELEsistheneedtodevelopqualityquestionswhichwillengagestudentsincriticalthinking.Questioninginitiatedthroughtheface-to-faceTELEcanleadtodiscussioninclasswhichiscontinuedonlinetofurtherstimulateinquiry,discussion,criticalthinking,andreflection.Inournursingprogramtheinclusionofcriticalthinkingandtechnologyintocoursesisnotalwayspurposefullyimplemented,beingsporadicattimesandlefttoindividualinstructors’prefer-ences.Withnospecificcoursesassociatedwithcriticalthinkingitisoftenpresumedincorporatedsomewhere,somehowinthepro-gram.Demonstrationofcriticalthinkingisanexpectationofnursingstudentsoftenwithoutexplicitinstructiononwhatitisorhowtodevelopsuchcapabilityinthemselvesleavingthemun-certainonwhattheyneedtoaccomplish.Similarly,educatorsseekingtoteachcriticalthinkingencounteravarietyofrecom-mendedstrategiesandarechallengedtodeterminebestmeansforinstructionandassessment(Blondy,2011;Chan,2013;Raymond-SeniukandProfetto-McGrath,2011).Theimportanceofincorpo-ratingtechnologyintocurriculahasbeennotedhoweverlackofconsensusandreluctancetotransformteachingpracticecanbeabarriertoadoptionofTELEs(DeGagne,Bisanar,Makowski,andNeumann,2012;WachiraandKeengwe,2011).3.ResearchstudyAresearchstudywasinitiatedwithacohortofnoviceunder-graduatenursingstudentsastheybegancourseswithnursing-specificcontentintheprogram.Amixedmethodsapproachwasimplementedtogainunderstandingofthestudentpositionondevelopingcriticalthinkingcapabilitiesandtheuseoftechnologytoenhancelearningexperiences.Thestudyblendedface-to-faceandonlineuseoftechnologyasenvironmentsforinquiry,discus-sion,andreflection.CriticalthinkinginstructionwasintroducedandalignedwithcoursecontentandrelatedtopracticeaswerethetypesofquestionsandpromptsgeneratedforusewiththeTELEs.Instructionalstrategieshaveinsomesituationsbeenimplementedbasedonwhateducatorsdeembeneficialforstudentlearning.Improvedunderstandingofstudentperspectivecanthereforesupporteducatorsindeliveringinstructionbettersuitedandusefultostudents,withpotentialforgreateruptakeandlearningbythestudents.Thisstudyaimedtodetermine:TowhatextentincorporatingexplicitcriticalthinkinginstructionandTELEscontributetonursingstudentdevelopmentofcriticalthinking?Fundamentally,totargetthestudentperspectiveon:a.Useofaclassroomresponsesystemasasupportingplatformforthedevelopmentofcriticalthinking.b.Potentialoftheonlinediscussionforumindevelopmentofcriticalthinking.c.Instructionalstrategiesimplementedtofacilitatedevelopmentofcriticalthinking.3.1.ResearchdesignAmixedmethodsapproachwaschosenforitsvaluewhenexaminingcomplexissuesinvolvingthehumanelementinte-gratingthestrengthsofquantitativeandqualitativeresearchtoacquiredeepmeaningfulunderstanding(ZhangandCreswell,2013).Inthisstudy,datawascollectedtocapturethelearningexperienceofthestudentswhilealsoassessingfordemonstrationofcriticalthinking.Findingsfromthesedifferentsourcesofdatahavecapacitytosubstantiateeachother,whilealsoidentifyingpotentialgapsandcontradictionsforfurtherstudy.Quantitativedatacollectionincluded:AcommerciallyavailablecriticalthinkingskillstestLikertscaleclose-endedquestionsintheend-of-termsurveyQualitativedatacollectionincluded:R.Swart/NurseEducationinPractice23(2017)30e3931

Open-endedquestionsintheend-of-termsurveyOnlinediscussionpostings3.2.TheCommunityofInquiryframeworkTheCommunityofInquiry(CoI)frameworkwasconsideredsuitableforthisstudyasitisfocusedonthecreationofaneduca-tionalexperienceconducivetolearninganddevelopingcriticalthinkingcapability(Garrison,2011;Swanetal.,2009;ToluandEvans,2013).Garrison(2011)initiallydevelopedtheCoItosup-porttechnology-enhancedlearningfoundedoncommunity,collaboration,constructivism,inquiry,andmetacognition.Estab-lishingaCoIwithintheface-to-faceTELEhasvalueforintroducingknowledgeandinitiatingdiscussionwhiletheonlineforumhaspotentialtostimulatemorereflective,focused,andthoughtfuldiscussion(Garrison,2011;NaberandWyatt,2014).Whilethein-classTELEfacilitates“quickthinking”andclassroomdiscussion,theonlineTELEpromotesmore“reflectivethinking”.GarrisonandVaughan(2008)identifytheonlineforumasbeneficialfordevel-opinggreaterthoughtandreflectionthroughwrittencommuni-cationandcircumventingtheneedforimmediateresponserequiredduringin-classdiscussions.Incorporatingbothin-classandonlineenvironmentsforlearningwouldthereforeprovidenursingstudentsopportunitytodevelopbothquickandreflectivethinking,bothbeneficialastheydeveloptheirnursingpractice.Garrison(2011)depictsthreeidentitiesportrayedthroughinteractionwithintheTELEsdsocialpresence,cognitivepresenceandteachingpresencedtofunctioninteractivelyintheCoItosupportdeliveryofalearningexperiencethatengagesstudentsininquiry,discussion,andreflectionleadingtocriticalthinking.Socialpresencesupportsengagementandinteractionwithotherswhilefosteringtrust,support,familiarity,andcollaborationtopromotecommunicationandadvancelearning.Pollard,Minor,andSwanson(2014)highlighttheinstructor’sportrayalofsocialpresencesignificantlyinfluencingthequalityofthelearning,supportingimportancefortheroleoftheinstructorinpositivestudentout-comes.Garrison(2011)andRichardsonetal.(2013)identifycognitivepresenceasinstrumentalinfacilitatingcriticalthinkingdevelopmentthroughmeaningfulinquiry,discussion,andreflection.Fortheinstructor,theteachingpresenceisespeciallyimportantbeingtheidentitywhichpurposefullydesigns,implements,facili-tates,andguidessocialandcognitivepresencewithintheCoI(Garrison,2011;Garrisonetal.,2010;HoslerandArend,2013).Aninquiry-basedapproachthatusesappropriatequestionsatsuitableperiodswithinin-classandonlineTELEstoassess,encourage,andevaluatestudentcomprehensionandknowledgecanbebeneficialinstimulatingcriticalthinking(DeLoachandGreenlaw,2007;HoslerandArend,2013;Kuetal.,2014;Pedrosa-de-Jesusetal.,2012;Richardsonetal.,2013;Tofadeetal.,2013).Inquiryfocusingontiming,content,type,andmannerofaskingcon-structedquestionscanbeusefulasaninstructionalstrategy.Thelevelofthinkingofquestionsaskedissignificantandshouldreflectthelevelofthinkingdesired;ifquestionsaskedreflectthehigherlevelsofthinking(Bloom’staxonomy)theyhavegreaterpotentialforelicitinghigherlevelthinkingfromstudents(Boswell,2005;Ertmeretal.,2011;Kimetal.,2012;Krathwohl,2002;Mayer,2002;Whiteley,2006).Gatheringdatafromthestudents’per-spectiveswastoinformonthetypeandcalibreofquestionstheyjudgedtobehelpfulindevelopingtheircriticalthinking.3.3.EthicalconsiderationsApprovalforthisresearchstudywasattainedfromtheuniversity’sConjointFacultiesResearchEthicsBoard(CFREB).AdditionalapprovalwasattainedfromtheAssociateDeanoftheUndergraduateNursingProgramandagreementandcollaborationwiththefacultyinstructorofthecourse.Thisaimedtoensurethattheresearcherhavenoinfluenceorinputintostudentgradesforthecourse.Thestudentswerefullyinformedastothepurposeoftheimplementedactivities,criticalthinkingskillstest,andsurveytoensureinformedconsentpriortotheinitiationofthestudy.Theywereassuredofconfidentialityandanonymityandmadeawarethattheresearchwastodeterminethesuitabilityoflearningwithtechnologyin-classandonlineanditsroleinthedevelopingofcriticalthinking.Studentswereinformedthattheirresponseswouldcontributetoimprovingthecourseandwouldnotbeafactorintheirgrade,thusestablishingthattherewasnoriskeitherwaytotheirchoosingtoparticipateornotparticipateinthestudy.Itwasreinforcedthattheresearcher’sroleintheclasswasstrictlyforresearchpurposes.Theinclusionandexclusioncriteriaestablishedforthisresearchstudyincludedallstudentsparticipatingintheclassasthelearningactivitieswereintegratedaspartoftheoverallcourse.Theend-of-termsurveywasanonymoustoprotectstudentprivacyandmaintainconfidentialityandstudentsassuredthattheywouldnotberevieweduntilafterthefinalgradesweredetermined.Followingthesefullydisclosedinstructionsandinformationaboutthestudy,completionofthesurveyimpliedconsenttoparticipate.Participantswerefurtherinformedthatresultsfromthetests,surveys,andlearningactivitieswouldbedisseminatedaspartofknowledgemobilizationactionstoinformeducatorsorpublishfindingsfromthestudy.Thefindingscouldbeemployedtoenhanceteachingandlearning,by(a)offeringanunderstandingofimple-mentinglearningtechnologiesineducation,(b)developingTELEs,and(c)engagingstudentsinthedevelopmentofcriticalthinkingcapability.3.4.StudyparticipantsAstheinstructionandlearningtechnologieswereimplementedaspartofthecourse,allofthe127enrolledstudentswereincludedinthestudy.Theclassof127registeredconsistedof11malesand116females,withthemajorityofstudentsinthe18e22-yearagerange.Thisidentifiedpopulationofsecond-yearnursingstudentsinahighereducationalsettingwerepurposefullysampled.Thesestudentswereintentionallychosenastheywereintheearlystagesoftheirnursingeducationandcouldprovidedatahelpfultobetterunderstandinghowtodevelopcriticalthinkinginundergraduatenursingstudents,alongsidehowtosupportandfacilitatestudentlearninginTELEs.Whileall127studentsparticipatedinthecourselearningac-tivitiesasimplementedthroughtheresearchstudy,onlythedatafromthestudentswhoprovidedconsentforuseoftheircontri-butionsandsurveyresponseswerecollectedforanalysis.Studentanonymitywasmaintainedtoprotectconsentingstudents.Ofthe127students,55(43%)providedconsenttoincludetheironlinediscussionpostingsfortheresearchstudy.Thesurveywasimple-mentedattheendoftheterm,beingtheendofthecourseandtheendoftheresearchstudy,yielding43(34%)completedsurveys.3.5.Datacollection3.5.1.CriticalthinkingskillstestAsinglegroupdesignwaschosenasthecohortofstudentsneededtoremainintactwiththecourseadditionsbeingappliedforthewholeclass,andacontrolgroupwasnotachievable.Admin-istrationofthecriticalthinkingskillstestfollowedtheexplicitR.Swart/NurseEducationinPractice23(2017)30e3932

criticalthinkinginstructionatthebeginningoftheterm,andwasaimedtodeterminepresenceofcriticalthinking.TheHealthSci-encesReasoningTest(HSRT)fromInsightAssessment(2013)wasthecriticalthinkingskillstestdeemedsuitableasitwaspresentedasa“validatedcriticalthinkingskillstest”focusedinthehealthsciencescontext(CoxandMcLaughlin,2014;InsightAssessment,2013).3.5.2.End-of-termsurveyTheend-of-termsurvey,aspresentedinTable1,wasimple-mentedasacross-sectionalsurveydesigninoneinstanceattheendofthecourse,andincludedbothclose-andopen-endedquestions.Thesurveyencompassed25close-endedLikertscalequestionscentredonfivefociaboutusingTELEstosupportstudentlearningandcriticalthinking.These25questionswerebasedona5-choicescale,rangingfromstronglydisagree,disagree,toagreeandstronglyagree,withaneutralchoiceinthemiddle.Thefouropen-endedquestionsaimedtogathermoredetailfromthestu-dentsontheirperspectiveoftechnologyandthetypesofquestionsbeneficialtotheirlearninganddevelopmentofcriticalthinking.3.5.3.OnlinediscussionpostingsDatacollectionfromtheonlinediscussionpostingswasongoingthroughouttheresearchstudy,andincludedallenrolledstudents.Whileallpostingswerecollected,onlythosefromstudentswhoprovidedconsenttousetheirmaterialswasanalyzedandincludedinthestudy.3.6.ScheduleoftheresearchstudyThestudentswereinformedoftheresearchstudyonthefirstdayofclasses,andaskedtocompletetheconsentformonlyafterbeingfullyinformedaboutthestudy.Table2providesanoutlineoftheimplementationofthestudyactivities,identifyingthesched-ulingofthecriticalthinkinginstructionandthefiverandomin-stancesintegratingtheuseofTopHatandtheonlinediscussionsontheDesire2Learn(D2L)system.Toinitiatethestudy,thestudentswereprovidedwithanintroductiontocriticalthinking,includingadefinition,anditsapplicationtotheirpresentandfuturelearning.Aninfusionapproachwasimplemented(Ennis,1989),entailingintentionalintegrationofcriticalthinkinginstructionintothecourse.Table1End-of-termsurvey.Questions1¼Stronglydisagree2¼Disagree3¼Neutral4¼Agree5¼Stronglyagree12345Howdoestheuseofaclassroomresponsesysteminfluencethedevelopmentofcriticalthinking?1.TheuseofTopHataddedtomysenseofparticipationinmylearning2.TheuseofTopHatincreasedmymotivationtoengageintheclassdiscussions3.UsingTopHatmademylearningmoreenjoyable4.IfeltmorelikeanactiveparticipantwhenIusedTopHat5.TopHatreducedtheanxietyassociatedwiththepossibilityofansweringquestionsincorrectlyHowdoescriticalthinkingdevelopwithinanonlinefacilitateddiscussionforum?6.OnlineDiscussionsaddedtomysenseofparticipationinmylearning7.OnlineDiscussionsincreasedmymotivationtoengageintheonlineclassdiscussions8.UsingOnlineDiscussionsmademylearningmoreenjoyable9.IfeltmorelikeanactiveparticipantwhenIusedtheOnlineDiscussions10.TheOnlineDiscussionshelpedmetoreflectontheinformationIlearnedandthediscussionsfromtheclassWhatarethelearners’preferenceforinstructionalstrategiesinsupportingtheirdevelopmentofcriticalthinking?11.Ienjoyedlearningwhentechnology(OnlineDiscussionsandTopHat)wasintegratedtoaddressavarietyoflearningneeds12.Technology(OnlineDiscussionsandTopHat),asusedinthiscourse,didnotinterferewithorinterruptteaching/learning13.Mylearningwasenhancedbythetechnology(OnlineDiscussionsandTopHat)usedinthiscourse14.Ibelievethatthetechnology(OnlineDiscussionsandTopHat)supportedmylearning15.Ifeltthatthetechnology(OnlineDiscussionsandTopHat)supportedthedeliveryoftheinformationInstructionalApproachAssistedintheConstructionofKnowledgeandCriticalThinking16.ThetypeofTopHatquestions,andresponse-discussionsrequiredmetousecriticalthinkingskills17.DiscussingtheresultsoftheTopHatquestionsstimulatedmythinkingtobuildmylevelofknowledge18.IpreferredtheClassroomResponseTechnology,TopHat,fordevelopingcriticalthinking19.TheleveloftheOnlineDiscussionpostingsandquestionsrequiredmetousecriticalthinkingskills20.ThetypeofquestionsusedforOnlineDiscussionsimprovedmyunderstandingoftheinformation21.IpreferredtheOnlineDiscussionsfordevelopingcriticalthinkingSatisfactionwithTechnology-EnhancedInstructionalApproach22.Ienjoyedthetypeoftechnology(OnlineDiscussionsandTopHat)usedinthiscourse23.Thevarietyoftechnology(OnlineDiscussionsandTopHat)engagedmeinlearning24.Iwouldliketoattendmoreclasses/sessionsthatutilizethesetypesoftechnology(OnlineDiscussionsandTopHat)tosupportmylearning25.Thetechnology(OnlineDiscussionsandTopHat)usedintheclassprovidedmetheabilitytoactivelyparticipateinmylearningOpen-endedshortanswerquestions:1.Whichlearningtechnology,TopHatorOnlineDiscussions,wasmorebeneficialforyourengagementindevelopingcriticalthinking?And,why?2.Wastheintegrationofbothtechnologiesmoreorlessbeneficialtoyourdevelopmentofcriticalthinking?WhyorWhynot?3.WhatquestionsusedduringTopHatstoodoutforyouinencouragingyoutocriticallythink?4.WhatquestionsorpromptsusedduringtheOnlineDiscussionsstoodoutforyouinencouragingyoutocriticallythink?R.Swart/NurseEducationinPractice23(2017)30e3933

InstructionwaspurposefullyassimilatedintothecoursetoteachaboutwhatcriticalthinkingishowinquirycansupportitsdevelopmentthecorrelationbetweencriticalthinkingandstudentlearningThefirstlecturethereforeincludedapresentationon“Devel-opingCriticalThinking”toassistunderstandingofcriticalthinkinganditsimportancetoprofessionalpractice:“Todevelopthecriticalthinkingyouneedthroughquestioningyourself,exploring,andreflecting,tobecapableofmakingdecisions,judgmentsthatwillenableyoutoprovideappropriate,effective,safecareforyourpatient.”TheAmericanPhilosophicalAssociation(APA)DelphiReport(1990)expertconsensusstatementoncriticalthinkingwasintroduced:“purposeful,reflectivejudgmentwhichmanifestsitselfinreasonedconsiderationofevidence,context,methods,standards,andconceptualizationindecidingwhattobelieveorwhattodo”.DiscussiontosupportstudentunderstandingincludedsharingcriticalthinkingdescriptionsbyEnnis(1991),Facione(2011),andPaulandElder(2008)asalternativerepresentationswithsimilaremphasisonsharedprinciplesincludingjudgment,decision-making,reflectionandself-regulation.Assuch,criticalthinkingemphasizesthemakingofhigh-quality,carefully-deliberatedde-cisionsbasedonallavailableinformation.Tohighlighttheimpor-tanceofthinkingcriticallyindecision-making,discussioncentredon“focused,thoughtful,reflectiveappraisalinconsideringtheev-idence,thecontext,thestandards,theprocesses,andpoliciesandproceduresindeterminingwhattobelieveandwhattodo.”Asecondpresentationon“Questioningtodevelopcriticalthinking,”fosteredtheimportanceofinquiryandreflectiontodevelopingcriticalthinking.Bloom’srevisedtaxonomy(Krathwohl,2002)wasintroducedtohelpstudentsunderstandthelevelofthinkingtheywereaimingtoachieveandthelevelofquestionstoaskthemselvestogeneratetheirownself-inquiry.TheCoIframedthelearningenvironmentfordevelopingthein-classandonlineTELEs,initiatingopenandapproachablecommu-nicationtowelcomeandengagestudentstoparticipateintheirlearning.Thestudentsneededtofeelsafe,supported,respected,andvalued,tobuildanenvironmentoftrustandconfidentialityimportantforsuccessfullearning(FanningandGaba,2007).Stu-dentswerealsoprovidedwithopportunitytogainfamiliarityandfeelsupportedwithusingthetechnologyplannedforthecourse,withtimetoaccessTopHatandtheonlinediscussionforumonD2Ltotrialsamplequestionsinclass.AligningwiththeCoIframeworkfortheTELEsincorporatedintothisstudy,theresearcherundertooktheteachingpresence,teachingthecriticalthinkingmaterial,developingappropriatequestions,andfacilitatingdiscussionforthetechnology-enhancedlearningactivities.Questionsgeneratedwereverifiedwiththecourseinstructortoensurealignmentwithcoursematerial.Expressingthesocialpresence,theresearcherneededtobewelcomingandresponsiveinstudentinteractions,andcommuni-cationexchanges.Providingcognitivepresence,theresearchergeneratedquestionsandresponsestoencourageinquiry,thinking,andreflection.ThequestionswerethusconstructedusingBloom’srevisedtaxonomy(Krathwohl,2002)asaguide,withtheaimthatthequestionsaimedtoelicithigherlevelthinking.4.StudyfindingsDataanalysiswasdependentontheresearchmethodanddatacollected.Theinclusionofacriticalthinkingskillstest,asurveywithopen-andclose-endedquestions,andonlinediscussionpostingsoffereddifferentformsofdatawhichbroughtvarietytothetypeofinformationattained.Quantitativeanalysisencom-passeddescriptiveanalysisofthesurveyresponsestotheclose-endedquestions.Qualitativecontentanalysisperformedwiththeonlinepostingsaimedtoidentifycriticalthinkingdemonstrationfromstudentresponsesanddiscussiononline.Thematicanalysiswascompletedwiththestudentresponsestotheopen-endedsurveyquestions,toidentifymainideasfromtheirperspectiveoncriticalthinkinginstructionanduseofTELEstosupporttheirlearning.4.1.Quantitativedataanalysis4.1.1.CriticalthinkingskillstestTheHSRTwasadministeredasanonlinetest,whichstudentscouldcompleteduringatwo-weekwindowatthebeginningoftheterm,followingthefirstlectureintroducingthemtocriticalthinking.Studentparticipationwas63%;80ofthe127enrolledstudentscompletedtheHSRT.Overall,64%(51of80)ofstudentscompletingthetestdemonstratedstrongcriticalthinkingcapabil-ities,withcapacitytoreasonandreflecttomakequalitydecisions(InsightAssessmentReport,2015).Table2Implementationscheduleofresearchstudy.LectureweekInstructionalstrategiesOneWeekBeforetheFirstLectureWelcometostudentsandintroductiontotheresearcherandthestudythroughemailInitiatethedevelopmentoftheCommunityofInquiryWeekOne:FirstLectureWelcomeandintroductionface-to-facein-class:DevelopthelearningenvironmentProvidestudentsinformationontheresearchstudyandhandedouttheStudentConsentFormsMSPowerPointpresentation“DevelopingCriticalThinking”Provideinstructionsonaccessingandcompletingtheonlinecriticalthinkingtest(HSRT)WeekTwoTrialuseofTopHatinclassTimeforstudentstoaccessandfindtheirgroupsonD2LWeekThreeMSPowerPointpresentation“Questioningtodevelopcriticalthinking”FirstsetofTopHatquestionsandfirstsubmissionofonlinepostingsWeekFourShortreminderof“Questioningtodevelopcriticalthinking”andengaginginself-inquirytodevelopowncriticalthinkingSecondsetofTopHatquestionsandsecondsubmissionofonlinepostingsWeekSevenMid-termExaminationWeekEightThirdsetofTopHatquestionsandthirdsubmissionofonlinepostingsWeekTenFourthsetofTopHatquestionsandfourthsubmissionofonlinepostingsWeekTwelveFifthsetofTopHatquestionsandfifthsubmissionofonlinepostingsWeekThirteen:LastClassForTheTermFinalend-of-termsurveydstudentscompleteandsubmittedatendofclassProvideinstructionsonaccessingandcompletingtheonlinecriticalthinkingtest(HSRT)R.Swart/NurseEducationinPractice23(2017)30e3934

4.1.2.End-of-termsurveyThe25close-endedLikertscalequestionsfromtheend-of-termsurveyyieldedordinaldata,suitablefordescriptivestatisticstodeterminemostfrequentresponses(BooneandBoone,2012).ThedatawerecountedandcompiledintoExcelspreadsheets,denotingnumberofresponsesforeachofthechoicesforeachquestion.ThenumberofresponsesweresummedforeachchoiceoftheLikertscale,andusingtheCOUNTIFSfunction,frequencydistributionsweregeneratedforeachofthequestions.Theresponsestothefouropen-endquestionsyieldedadditionalinsightintostudents’points-of-view,enablingthemtousetheirownwordsinresponding.4.1.2.1.Impactofaclassroomresponsesystem,TopHat,onthedevelopmentofstudentcriticalthinking.With75%(32of43)ofrespondentsindicatingactiveparticipationintheirlearning,and74%(31of43)findingitsusehelpfulforanonymityandreducinganxietywhenrespondingtoquestionsinclass,thestudentsconsideredtheuseofTopHattobepositiveinhelpingthemdevelopcriticalthinking.Theyappreciatedbeing“abletodiscussasaclasstheanswerseveryoneprovided,”andnotedTopHatasa“greatwaytotestmyknowledge.”4.1.2.2.Developmentofstudentcriticalthinkingwithinanonlinediscussionforum.Theonlinediscussionswereconsideredhelpfulinpromotingreflectiononclasscontentby60%(26of43)ofthestudents,with50%(22of43)notingenhancedparticipationintheirlearning.Studentsnotedthat:“Thequestionsdopromptmetoreflectonthecourseinformation.”“Bythinkingandreflectingtogetheraboutthequestionsandconnections[they]shapedmyunderstandingandenhancedmycriticalthinking.”“They[questions]allowmetopullfromtheknowledgeIlearnedinclassandapplythemtoquestionsindiscussionformat.”Thestudentsviewedtheonlineforumasopportunitiesforfurtherdiscussion,thinking,andcollaborativelearning.Theyvaluedtheadditionaltimeandaplacetodiscussissueswithotherstudentsonline:“Allowedmetoseewhatotherswerethinkingandallowed[me]tofurtherdevelopmyunderstandingofthetopic.”“Ilikehowthediscussionsallowustospeakwithothersintheclassandseehowopinionsdifferamongourclassmates.Ialsolikehowthediscussionquestionsletusthinkaboutandreviewtopicsthatwe’vediscussedinclass.”4.1.2.3.Studentperspectiveontechnologyintegrationintoinstruc-tiontofostercriticalthinking.65%(28of43)ofthestudentsconsideredthathavingbothtechnologiesinthecoursewasbene-ficialtotheirdevelopmentofcriticalthinking.Theyappreciatedhaving“differentmethodsofreinforcingmylearning”;and“bothtechnologiesencouragedparticipationandcriticalthinking.”Ofthe43students,54%(23)agreedthatincorporationoftwotypesoftechnologymetthedifferentpreferencesandlearningneedsofstudents.Thiswasfurthernotedwithsomestudents(58%[25of43])statingpreferencefortheclassroomresponsesystem,TopHat,andothers(40%[17of43])preferringtheonlinediscus-sionstosupporttheirlearning:Onestudentsharedthat“IreallylikedusingTopHat.Itwasagreatwaytotestmyknowledge.”Ontheotherhand,anotherstudentnotedthat“onlinediscus-sionsweremorebeneficialbecauseIhadtocriticallythinkaboutthetopicandcreateacohesiveresponse.Itrequiredmetothinkaboutthetopicandeverything[that]influencesthetopic.”SomestudentsnotedpreferenceforTopHatasit“wasdoneinclasstimeanditcoveredwhatwejustlearned.”Otherswhopreferredtheonlinediscussionsvaluedthat“wehadtimetoprocessideasandthoughtsinamorecriticalmanner.”4.1.2.4.Studentperspectiveontheinstructionalapproachesusedtofacilitatetheircriticalthinking.Ofthestudentswhocompletedthesurvey,93%(40of43)valuedthetypesofquestionsandresponsepromptsgeneratedwithinthein-classandonlineTELEsinfacili-tatingtheirdevelopmentofcriticalthinking.Fortheclassroomresponsesystem,TopHat,thestudentsidentifiedpreferencesforquestionswithmultiplecorrectanswers,andquestionswhichappliedcoursecontenttorealpracticesituations.Thestudentsappreciatedthesequestionsinencouragingdiscussionandfosteringdeeperthinking,with81%(35of43)ofthemnotingthatdiscussionfollowingencouragedthinkingandknowledgebuilding.Withtheonlinediscussions,thequestionsthatstudentsdeemedbeneficialinvolvedbeingaskedtoincorporatetheirpersonalperspective,applycourseknowledge,andincorporatepracticescenarios.Astudentsummarizedas:Ilikehowthediscussionquestionsplaceusindifferentsitua-tionsandinthatway,forcesustocombineboththelecturematerialtoreallife.Thishelpscementwhatwelearnedbeforehandandinourreadingsbymakingitmorepersonalandthus,moremeaningful.4.1.2.5.Studentsatisfactionwiththeuseoftechnologytoenhancetheirlearningenvironment.Ofthe43responses,70%(30)ofthestudentsfoundtheuseoftechnologydidnotinterferewiththeteachingandlearningoftheclass.Themajorityview,68%(29of43)ofthestudents,expressedthatthetechnologysupportedthede-liveryoftheinformation,supporting(58%[25of43])andenhancing(56%[24of43])theirlearninginthecourse.4.2.Qualitativedataanalysis4.2.1.End-of-termsurveyTheresponsesattainedfromtheopen-endedsurveyquestionsyieldeddataamenabletothematicanalysis.Recognizedforflexi-bilityinanalyzingtextdata,thematicanalysisisconsideredusefulfororganizingopen-endedsurveyresponsesinrecognizingpat-ternsforanalysisanddescriptionuponrepeatedperusal(BraunandClarke,2006).MainideaswereidentifiedbythestudentssupportingtheLikertscaleresponsesandprovidingmoredetailedcomprehension:Benefitsofusingin-classTELE:multiplechoiceformataspreparationforexaminations;completioninclasstime;andgeneratingclassroomdiscussion.Benefitsofusingonlinediscussions:provisionoftimetoprocessandreflect;allowingfordeeperthinkingtounderstand;andfacilitatingapplicationandconnection-makingofcoursematerials.R.Swart/NurseEducationinPractice23(2017)30e3935

Benefitsofusingbothin-classandonlineTELEs:engagementandactiveparticipation;applicationandmakingconnectionsofcoursecontent;andreflection,criticalthinking,andunder-standingofcoursematerials.Thestudentsrevealedemergingself-awarenessandperceptionofcriticalthinkingasidentifiedinthefollowingideasnotedfromtheirsurveyresponses:Timeneededtoprocesslearning.Deeperthinkingtogainunderstanding.Applyinglearningandmakingconnections.Personalizinglearningasameansbywhichtheyweredevel-opingcriticalthinking.Theynotedthat“itrequiredmoreprocessingtimetocriticallythink”and“timetoprocessideasandthoughts,”recognizingthevalueoftimeindevelopingcriticalthinking.Theyshared“helpingmefurtherunderstandthecontent”and“itgetsusthinking,especiallyonmaterialthatrelatestoclass,”recognizingthatgain-ingunderstandingwashelpfultotheirdevelopingofcriticalthinking.Statementsof“theyhelptoapplytheconceptsandlearning,andconsiderreallifesituations”and“[this]helpedmereflectonwhatIhavelearnedinclassandapplyittothequestionasked,”revealedawarenessfortheusefulnessofapplicationandconnectionsmadeinlearningtodevelopingcriticalthinkingcapability.Studentsidentifiedquestionswhichaskedthemtoincorporatetheirper-sonalperspectivesastypesofquestionsandpromptshelpfultotheirdevelopmentofcriticalthinking:“discuss[ing]personalexperiencewiththeknowledgewehadgainedfromclass…reallyhelpedmerememberthecontentinthefuture.”Studentsrecognizedtheneedtothink,reflect,connect,apply,anddeterminewhybeforecommittingtoadecision.Theyexpressedcomprehensionofcriticalthinkingwithresponsessuchas“beingabletoformfullthoughtsandreasonstosupportthem”;“helpedmereflect…andapplyit”;and“requiredyoutothinkoutanswersandprovewhyitwasthebestone.”4.2.2.OnlinediscussionpostingsQualitativecontentanalysisofonlinediscussionpostingsfacil-itatesunderstandingofthemeaningofthetextdata,usingpre-determinedcodingschemesrelevanttothecontenttoestablishinitialthemesforcoding(Schrire,2006;HsiehandShannon,2005).Theuseofthreemethodsofcontentanalysishasbeenrecom-mendedfortheassessmentforcriticalthinkingtofacilitateconfirmationofthecapabilityandpotentiallycontributetoimprovingtriangulation(Schrire,2006).Consequently,thePracticalInquiryModel(PIM)(Garrison,2011),Bloom’srevisedtaxonomy(Krathwohl,2002),andMason’s(2000)criticalthinkingindicatorswereusedtoidentifycriticalthinkinginthetextualcontentofthediscussionpostsassummarizedinTable3.Garrison(2011)proposedtheuseofthePIMforassessingcriticalthinkingandasameanstodeterminecognitivepresencewithintheCoI.ThePIMencompassesfourphasesasdescribedinTable3.Corich,Kinshuk,andHunt(2004)demonstratedtheusefulnessofthePIMinqualitativecontentanalysisofonlinediscussionpost-ings.Similarly,Schrire(2006)usedthePIMwiththecognitivelevelsofBloom’srevisedtaxonomyinassessmentofonlinedis-cussionpostsforcriticalthinking.Newman,Webb,andCochrane(2004)employedMason’s(2000)criticalthinkingindicator-questionsasameanstoassessforcriticalthinkingandtodemon-stratethevalueofalearningactivityimplementedinacourse.Atotalof385onlinediscussionpostingswereanalyzedbyeachofthethreemodels,andrevealedstudents’demonstrationsofcriticalthinking.Themajorityofstudentposts(79%)exemplifiedtheexplorationphaseofthePIM,with21%integration,revealingcriticalthinkingcapability(Garrison,2011;Schrire,2006).Thelevelsofanalyzing(55%ofposts)andevaluating(23%ofposts)fromBloom’srevisedtaxonomyweredemonstratedandsupportedcriticalthinking(Krathwohl,2002).BasedontheindicatorsnotedbyMason(2000),thestudentsdisplayedcriticalthinking,inreferringtocourselearning(55%ofposts)andmovingtowardsnewideasandperspectives(27%ofposts).5.DiscussionThefindingsfromthisstudyrevealthatcertainfactorsarebeneficialtostudentdevelopmentofcriticalthinking,andimpor-tanttobeincludedininstruction:Intentionalintegrationofcriticalthinkinginstructionwithexplicitdescriptionofwhatcriticalthinkingisandspecificcorrelationtoprofessionalpractice.PurposefulincorporationofdifferentTELEsalignedwithcoursecontent.Aimofdiscussionsandquestionsatthelevelofthinkingdesiredinthestudents.Creationofquestionstoconnectcoursematerialtopracticesituations.Analysisofthedatacollectedfromstudentsfollowingexplicitcriticalthinkinginstructiondemonstratedtheirunderstandingandexpressionofcriticalthinking.Assuch,resultsfromthecriticalthinkingskillstestrevealedastrongpresenceofcriticalthinkinginthemajorityofstudentswhocompletedthetest.Students’criticalthinkingcapabilitywasalsodemonstratedintheirdiscussionpostings,asevidencedfromthecontentanalysisofthepostings,andsubstantiatedbytheuseofthreemodelstoanalyzethecon-tent.Thestudentsalsoexpressedself-awarenessofcriticalthinking,asshownfromthethematicanalysisoftheirresponsestothesurveyquestions,usinglanguagedemonstratingunderstandingofcriticalthinking.AsnotedbyAkyol,Garrison&Ozden,(2009)theperceptionofstudentsoftheirpersonalcapabilitiesisimportanttotheirunderstandinganddevelopmentofcriticalthinking.Withpotentialtosupportinnovationanduseoftechnologyineducation,thestudentsidentifiedthevalueofTELEsfocusedtosupportinquiry.Accordingly,theyalsoviewedhavingvarietywithtwoTELEswashelpfulforprovidingdifferentmeansoflearning.Thestudentsnotedquestionswhichrequiredthemtoapply,personalize,andconnectcoursecontenttoreallifepracticeasusefultodevelopingcriticalthinking.Thisreinforcestheimpor-tanceofbeingconsciouswhencreatingquestionsthattheydostimulatestudentstothinkatahigherlevel,ifanobjectiveistodevelopcriticalthinking(Tofadeetal.,2013).Inaddition,Richardsonetal.(2013)likewiserecommendusingquestionswhichentailmakingconnections,constructingmeaning,creatingsolutions,andincorporatingpersonalinsighttofacilitateattain-mentofa“highercriticalthinkinglevel”(p.210).Whilethefindingsarelimitedtothiscohortofundergraduatenursingstudents,thereispotentialfortheinstructionalstrategiesandtechnology-enhancedlearningidentifiedasbeneficialtodevelopingcriticalthinkingtoinformothersimilarsituations.Furtherresearchwithapplicationofthefindingstodifferentstu-dentsituationsandprogramsisneeded.R.Swart/NurseEducationinPractice23(2017)30e3936

6.ConclusionsFromthisresearchstudy,informationwasgatheredfromthestudents’perspectivestoidentifyinstructionalapproachestheydeemedbeneficialtostudentdevelopmentofcriticalthinking.Thestudents’commentsshowedtheyappreciatedthequestionsdevelopedfortheteachingbecausetheywererelevanttocoursecontentandprofessionalpractice.Thishighlightstheimportanceofinquiryfoundedondomainknowledgeincriticalthinkingdevel-opment.Questionswhichaskedthemtoapplytheirlearningandwhichpersonalizedthelearningwerenotedtobehelpfultodevelopingtheircriticalthinking.Creatingquestionsaimedathigherlevelsofthinkinggenerateddemonstrationsofthatlevelofthinkingintheonlinediscussionpostings.Manystudentsidentifiedthattheinclusionoftechnologytoenhancetheirlearningwasappreciated,aswashavingthevarietyofin-classandonlinetechnologytofacilitatetheirdevelopmentofcriticalthinking.Thestudyfindingsdemonstratedthatappropri-atelyintegratingTELEstoachievespecificobjectivesintoanursingtheoryclasshasthepotentialtosupportthestudents’developmentofcriticalthinking.Designingacoursewithsuchcomponentswillhelpstudentsachievetheselearningoutcomes.AcknowledgementsMuchappreciationtoDr.JenniferLock,UCalgaryWerklundSchoolofEducationforhersupportandguidancethroughthestudy.ReferencesAkyol,Z.,Garrison,D.R.,Ozden,M.Y.,2009.Developmentofacommunityofinquiryinonlineandblendedlearningcontexts.ProcediaSoc.Behav.Sci.1(2009),1834e1838.http://dx.doi.org/10.1016/j.sbspro.2009.01.324.Anthis,K.,2011.Isittheclicker,orisitthequestion?Untanglingtheeffectsofstudentresponsesystemuse.Teach.Psychol.38(3),189e193.http://dx.doi.org/10.1177/0098628311411895.Arend,B.,2009.Encouragingcriticalthinkinginonlinethreadeddiscussions.J.Educ.Online6(1),1e23.Retrievedfrom.http://www.thejeo.com.Beatty,I.D.,Gerace,W.J.,Leonard,W.J.,Dufresne,R.J.,2006.Designingeffectivequestionsforclassroomresponsesystemteaching.Am.J.Phys.74(1),31e39.http://dx.doi.org/10.1119/1.2121753.Behar-Horenstein,L.,Niu,L.,2011.Teachingcriticalthinkingskillsinhigheredu-cation:areviewoftheliterature.J.Coll.Teach.Learn.8(2),25e42.http://dx.doi.org/10.19030/tlc.v8i2.3554.Benjamin,R.,Elliot,S.,Klein,S.,Patterson,J.,Steedle,J.,Zahner,D.,2013.TheCaseforCritical-thinkingSkillsandPerformanceAssessment.RetrievedDecember27,2015from.http://cae.org.Bensley,D.A.,2011.Rulesforreasoningrevisited:towardascientificconceptionofcriticalthinking.In:Horvath,C.P.,Forte,J.M.(Eds.),CriticalThinking:EducationinaCompetitiveandGlobalizingWorld.NovaSciencePublishing,Inc.,NewYork,pp.1e36.Berkow,S.,Virkstis,K.,Stewart,J.,Aronson,S.,Donohue,M.,2011.Assessingindi-vidualfrontlinenursecriticalthinking.J.Nurs.Adm.41(4),168e171.http://dx.doi.org/10.1097/NNA.0b013e3182118528.Blondy,L.C.,2011.Measurementandcomparisonofnursingfacultymembers’criticalthinkingskills.West.J.Nurs.Res.33(2),180e195.http://dx.doi.org/10.1177/0193945910381596.Boone,H.N.,Boone,D.A.,2012.AnalyzingLikertdata.J.Ext.50,2.ArticleNumber2TOT2.Retrievedfrom.http://www.joe.org.Borglin,G.,Fagerstrom,C.,2012.Nursingstudents’understandingofcriticalthinkingandappraisalandacademicwriting:adescriptive,qualitativestudy.NurseEduc.Pract.12,356e360.http://dx.doi.org/10.1016/j.nepr.2012.04.009.Boswell,C.,2005.Theartofquestioning:improvingcriticalthinking.Annu.Rev.Nurs.Educ.4291e4304.Retrievedfrom.http://connection.ebscohost.com.Brady,M.,Seli,H.,Rosenthal,J.,2013.Metacognitionandtheinfluenceofpollingsystems:howdoclickerscomparewithlowtechnologysystems.Educ.Technol.Res.Dev.61,885e902.http://dx.doi.org/10.1007/s11423-013-9318-1.Braun,V.,Clarke,V.,2006.Usingthematicanalysisinpsychology.Qual.Res.Psychol.3(2),77e101.http://dx.doi.org/10.1191/1478088706qp063oa.Brown,B.,2013.Technology-enhancedlearningenvironment.In:Richey,R.C.(Ed.),EncyclopediaofTerminologyforEducationalCommunicationsandTechnology.Springer,NewYork,p.304.Burrell,L.A.,2014.Integratingcriticalthinkingstrategiesintonursingcurricula.Teach.Learn.Nurs.9,53e58.http://dx.doi.org/10.1016/j.teln.2013.12.005.Chan,Z.C.Y.,2013.Asystematicreviewofcriticalthinkinginnursingeducation.NurseEduc.Today33,236e240.http://dx.doi.org/10.1016/j.nedt.2013.01.007.Cheong,C.M.,Cheung,W.S.,2008.Onlinediscussionandcriticalthinkingskills:acasestudyinaSingaporesecondaryschool.Australas.J.Educ.Technol.24(5),556e573.RetrievedDecember24,2015from.http://ascilite.org.au.Corich,Kinshuk,S.,Hunt,L.M.,2004.AssessingDiscussionForumParticipation:inSearchofQuality.RetrievedDecember24,2015from.http://www.itdl.org.Cox,W.C.,McLaughlin,J.E.,2014.Associationofhealthsciencesreasoningtestscoreswithacademicandexperientialperformance.Am.J.Pharm.Educ.78(4).http://dx.doi.org/10.5688/ajpe78473.DeBourgh,G.A.,2008.Useofclassroom“clickers”topromoteacquisitionofadvancedreasoningskills.NurseEduc.Pract.8(2008),76e87.http://dx.doi.org/10.1016/j.nepr.2007.02.002.DeLoach,S.B.,Greenlaw,S.,2007.Effectivelymoderatingelectronicdiscussions.J.Econ.Educ.38(4),419-421,423,425-434.http://dx.doi.org/10.3200/jece.38.4.419-434.Dennen,V.P.,2013.Technologicalcommunication.In:Richey,R.C.(Ed.),Encyclo-pediaofTerminologyforEducationalCommunicationsandTechnology.Springer,NewYork,p.297.Dziuban,C.D.,Hartman,J.L.,Cavanagh,T.B.,Moskal,P.D.,2011.Blendedcoursesasdriversofinstitutionaltransformation.In:Kitchenham,A.(Ed.),BlendedLearningacrossDisciplines:ModelsforImplementation.InformationScienceReference,Hershey,pp.17e37.Ennis,R.H.,1989.Criticalthinkingandsubjectspecificity:clarificationandneededresearch.Educ.Res.18(3),4e10.RetrievedDecember24,2015from.http://www.jstor.org.Ennis,R.H.,1991.Criticalthinking:astreamlinedconception.Teach.Philos.14(1),5e24.RetrievedDecember24,2015from.http://www.criticalthinking.net.Ertmer,P.A.,Sadaf,A.,Ertmer,D.J.,2011.Student-contentinteractionsinonlinecourses:theroleofquestionpromptsinfacilitatinghigher-levelengagementwithcoursecontent.J.Comput.High.Educ.23(2e3),157e186.http://dx.doi.org/10.1007/s12528-011-9047-6.Facione,P.A.,2011.CriticalThinking:whatitIsandWhyitCounts.InsightAssess-ment.TheCaliforniaAcademicPress,California.Fanning,R.M.,Gaba,D.M.,2007.Theroleofdebriefinginsimulation-basedlearning.Simul.Healthc.2(2),115e125.Retrievedfrom.http://multibriefs.com.Fero,L.J.,Wisberger,C.M.,Wesmiller,S.W.,Zullo,T.G.,Hoffman,L.A.,2009.Criticalthinkingabilityofnewgraduateandexperiencednurses.J.Adv.Nurs.65(1),Table3Summaryofcontentanalysismethods.Practicalinquirymodel(PIM)descriptors(Garrison,2011)Bloom’srevisedtaxonomydescriptors(Krathwohl,2002)Criticalthinkingindicators(Mason,2000)Triggeringevent:QuestioningormessagingtoencouragediscussionRemembering:RecallingrelevantinformationUnderstanding:Explainingideasorconcepts;makingsenseofwhatyouhavelearnedDotheparticipantsbuildonpreviousmessages?Exploration:AwarenessofaspectsoftheissueorproblemApplying:Usinginformationinanotherfamiliarsituation;usingknowledgegainedinnewwaysAnalyzing:Breakinginformationintopartstoexploreunderstandingsandrelationships;breakingconceptintopartstounderstandhoweachpartisrelatedtooneanotherDotheydrawontheirownexperience?Dotheyrefertocoursematerial?Integration:Reflectionanddeliberationtoconnectvariousinformationalsources,developingpotentialresolutions/proposals/conceptionsEvaluating:Justifyingadecisionorcourseofaction;makingjudgmentsbasedonasetofguidelinesDotheyrefertorelevantmaterialoutsidethecourse?Resolution:Commitmenttosolutionwithvicariousapplication/testinginrealworldCreating:Generatingnewideas,products,orwaysofviewingthings;puttinginformationtogetherinaninnovativewayDotheyinitiatenewideasfordiscussion?R.Swart/NurseEducationinPractice23(2017)30e3937

139e148.http://dx.doi.org/10.1111/j.1365-2648.2008.04834.x.Flores,K.L.,Matkin,G.S.,Burbach,M.E.,Quinn,C.E.,Harding,H.,2012.Deficientcriticalthinkingskillsamongcollegegraduates:implicationsforleadership.Educ.PhilosophyTheory44(2),212e230.http://dx.doi.org/10.1111/j.1469-5812.2010.00672.x.DeGagne,J.C.,Bisanar,W.A.,Makowski,J.T.,Neumann,J.L.,2012.IntegratinginformaticsintotheBSNcurriculum:areviewoftheliterature.NurseEduc.Today32,675e682.http://dx.doi.org/10.1016/j.nedt.2011.09.003.Garrison,D.R.,2011.E-learninginthe21stCentury:aFrameworkforResearchandPractice.RoutledgeTaylorandFrancisGroup,NewYork.Garrison,D.R.,2013.Blendedlearning.In:Richey,R.C.(Ed.),EncyclopediaofTer-minologyforEducationalCommunicationsandTechnology.Springer,NewYork,p.23.Garrison,D.R.,Vaughan,N.,2008.BlendedLearninginHigherEducation:Frame-work,Principles,andGuidelines.JohnWileyandSons,Inc.,SanFrancisco.Garrison,D.R.,Cleveland-Innes,M.,Fung,T.S.,2010.Exploringcausalrelationshipsamongteaching,cognitiveandsocialpresence:studentperceptionsofthecommunityofinquiryframework.InternetHigh.Educ.13,31e36.http://dx.doi.org/10.1016/j.iheduc.2009.10.002.Heijltjes,A.,VanGog,T.,Paas,F.,2014.Improvingstudents’criticalthinking:empiricalsupportforexplicitinstructionscombinedwithpractice.Appl.Cogn.Psychol.28(4),518e530.http://dx.doi.org/10.1002/acp.3025.Hosler,K.A.,Arend,B.D.,2013.Strategiesandprinciplestodevelopcognitivepresenceinonlinediscussions.In:Akyol,Z.,Garrison,D.R.(Eds.),EducationalCommunitiesofInquiry:TheoreticalFramework,Research,andPractice.In-formationScienceReference,Hershey,pp.148e167.Hsieh,H.-F.,Shannon,S.E.,2005.Threeapproachestoqualitativecontentanalysis.Qual.HealthRes.15(9),1277e1288.http://dx.doi.org/10.1177/1049732305276687.Hunter,S.,Pitt,V.,Croce,N.,Roche,J.,2014.Criticalthinkingskillsofundergraduatenursingstudents:descriptionanddemographicpredictors.NurseEduc.Today34,809e814.http://dx.doi.org/10.1016/j.nedt.2013.08.005.InsightAssessment,2013.HealthSciencesReasoningTest(HSRT).TheCaliforniaAcademicPressLLC,SanJose.Retrievedfrom.http://www.insightassessment.com.InsightAssessment,2015.TheHealthSciencesReasoningTestUserManual.Retrievedfrom.http://www.insightassessment.com.Kim,M.-K.,Patel,R.A.,Uchizono,J.A.,Beck,L.,2012.IncorporationofBloom’sTax-onomyintomultiple-choiceexaminationquestionsforapharmacotherapeuticscourse.Am.J.Pharm.Educ.76(6),1e8.http://dx.doi.org/10.5688/aipe766114.Kim,C.M.,Kim,M.K.,Lee,C.,Spector,J.M.,DeMeester,K.,2013.Teacherbeliefsandtechnologyintegration.Teach.Teach.Educ.29,76e85.http://dx.doi.org/10.1016/j.tate.2012.08.005.Korkmaz,O.,Karakus,U.,2009.Theimpactofblendedlearningmodelonstudentattitudestowardsgeographycourseandtheircriticalthinkingdispositionsandlevels.TurkishOnlineJ.Educ.Technol.8(4),51e63.RetrievedDecember24,2015from.http://www.tojet.net/articles/v8i4/845.pdf.Krathwohl,D.R.,2002.ArevisionofBloom’sTaxonomy:anoverview.TheoryintoPract.41(4),212e218.RetrievedDecember24,2015from.http://www.unco.edu.Krupat,E.,Sprague,J.M.,Wolpaw,D.,Haidet,P.,Hatem,D.,O’Brien,B.,2011.Thinkingcriticallyaboutcriticalthinking:ability,dispositionorboth?Med.Educ.45,625e635.http://dx.doi.org/10.1111/j.1365-2923.2010.03910.x.Ku,K.Y.L.,Ho,I.T.,Hau,K.-T.,Lai,E.C.M.,2014.Integratingdirectandinquiry-basedinstructionintheteachingofcriticalthinking:aninterventionstudy.Instr.Sci.42,251e269.http://dx.doi.org/10.1007/s11251-013-9279-0.Lai,K.,2012.Assessingparticipationskills:onlinediscussionswithpeers.Assess.Eval.High.Educ.37(8),933e947.http://dx.doi.org/10.1080/02602938.2011.590878.Lavin,A.M.,Korte,L.,Davies,T.L.,2010.Theimpactofclassroomtechnologyonstudentbehavior.J.Technol.Res.1e13.Retrievedfrom.http://www.aabri.com.Marin,L.,Halpern,D.F.,2011.Pedagogyfordevelopingcriticalthinkinginadoles-cents:explicitinstructionproducesgreatestgains.Think.Ski.Creativity6,1e13.http://dx.doi.org/10.1016/j.tsc.2010.08.002.Mason,R.,2000.Fromdistanceeducationtoonlineeducation.InternetHigh.Educ.3(1e2),63e74.http://dx.doi.org/10.1016/S1096-7516(00)00033-6.Mayer,R.E.,2002.Roteversusmeaningfullearning.TheoryintoPract.41(4),226e232.RetrievedDecember24,2015from.http://www.unco.edu.Mollborn,S.,Hoekstra,A.,2010.“Ameetingofminds”:usingClickersforcriticalthinkinganddiscussioninlargesociologyclasses.Teach.Sociol.38(1),18e27.http://dx.doi.org/10.1177/0092055X0935.Morrall,P.,Goodman,B.,2012.Criticalthinkingnurseeducationanduniversities:somethoughtsoncurrentissuesandimplicationsfornursingpractice.NurseEduc.Today33,935e937.http://dx.doi.org/10.1016/j.nedt.2012.11.011.Naber,J.,Wyatt,T.H.,2014.Theeffectofreflectivewritinginterventionsonthecriticalthinkingskillsanddispositionsofbaccalaureatenursingstudents.NurseEduc.Today34(1),67e72.http://dx.doi.org/10.1016/j.nedt.2013.04.002.Newman,D.R.,Webb,B.,Cochrane,C.,2004.AContentAnalysisMethodtoMeasureCriticalThinkinginFace-to-faceandComputerSupportedGroupLearning.RetrievedDecember24,2015from.http://www.umsl.edu.Oh,E.,Reeves,T.C.,2014.Generationaldifferencesandtheintegrationoftechnologyinlearning,instruction,andperformance.In:Bishop,M.J.,Spector,J.M.(Eds.),HandbookofResearchonEducationalCommunicationsandTechnology.SpringerScienceþBusinessMedia,NewYork,pp.819e828.Paul,R.,Elder,L.,2008.TheMiniatureGuidetoCriticalThinking:ConceptsandTools.RetrievedDecember24,2015from.http://www.criticalthinking.org.Pedrosa-de-Jesus,H.,Lopes,B.,Moreira,A.,Watts,M.,2012.Contextsforques-tioning:twozonesofteachingandlearninginundergraduatescience.High.Educ.64,557e571.http://dx.doi.org/10.1007/s10734-012-9512-9.Pisutova-Gerber,K.,Malovicova,J.,2009.Criticalandhigherorderthinkinginon-linethreadeddiscussionsintheSlovakcontext.Int.Rev.Res.OpenDistanceLearn.10(1).Retrievedfrom.http://www.irrodl.org.Pollard,H.,Minor,M.,Swanson,A.,2014.InstructorsocialpresencewithintheCommunityofInquiryframeworkanditsimpactonclassroomcommunityandthelearningenvironment.OnlineJ.DistanceLearn.Adm.17(2).RetrievedDecember24,2015from.http://www.westga.edu.Raymond-Seniuk,C.,Profetto-McGrath,J.,2011.Canonelearntothinkcritically?-Aphilosophicalexploration.OpenNurs.J.5,45e51.http://dx.doi.org/10.2174/1874434601105010045.Richardson,J.C.,Ice,P.,2010.Investigatingstudents’levelofcriticalthinkingacrossinstructionalstrategiesinonlinediscussions.InternetHigh.Educ.13(2010),52e59.http://dx.doi.org/10.1016/j.iheduc.2009.10.009.Richardson,J.C.,Sadaf,A.,Ertmer,P.A.,2013.Relationshipbetweentypesofquestionpromptsandcriticalthinkinginonlinediscussions.In:Akyol,Z.,Garrison,D.R.(Eds.),EducationalCommunitiesofInquiry:TheoreticalFramework,Research,andPractice.InformationScienceReference,Hershey,pp.197e222.Romeo,E.M.,2013.Thepredictiveabilityofcriticalthinking,nursingGPA,andSATscoresonfirst-timeNCLEX-RNperformance.Nurs.Educ.Perspect.34(4),248e253.RetrievedDecember24,2015from.http://www.ncbi.nlm.nih.gov.Roschelle,J.,Bakia,M.,Toyama,Y.,Patton,C.,2011.Eightissuesforlearningscien-tistsabouteducationandtheeconomy.J.Learn.Sci.20,3e49.http://dx.doi.org/10.1080/10508406.2011.528318.Saade,R.G.,Morin,D.,Thomas,J.D.E.,2012.Criticalthinkingine-learningenvi-ronments.Comput.Hum.Behav.28,1608e1617.http://dx.doi.org/10.1016/j.chb.2012.03.025.Schrire,S.,2006.Knowledgebuildinginasynchronousdiscussiongroups:goingbeyondquantitativeanalysis.Comput.Educ.46,49e70.http://dx.doi.org/10.1016/j.compedu.2005.04.006.Shieh,R.,2010.Acasestudyofconstructivistinstructionalstrategiesforadulton-linelearning.Br.J.Educ.Technol.41(5),706e720.http://dx.doi.org/10.1111/j.1467-8535.2009.00965.x.Shieh,R.S.,Chang,W.,2013.Implementingtheinteractiveresponsesysteminahighschoolphysicscontext:interventionandreflections.Australas.J.Educ.Technol.29(5),748e761.http://dx.doi.org/10.14742/ajet.331.Siau,K.,Sheng,H.,Nah,F.F.-H.,2006.Useofaclassroomresponsesystemtoenhanceclassroominteractivity.IEEETrans.Educ.49(3),398e403.http://dx.doi.org/10.1109/TE.2006.879802.Snodgrass,S.,2011.Wikiactivitiesinblendedlearningforhealthprofessionalstudents:enhancingcriticalthinkingandclinicalreasoningskills.Australas.J.Educ.Technol.27(4),563e580.http://dx.doi.org/10.14742/ajet.938.Spiliotopoulos,V.,2011.Towardsatechnology-enhanceduniversityeducation.In:Kitchenham,A.(Ed.),BlendedLearningacrossDisciplines:ModelsforImple-mentation.InformationScienceReference,Hershey,pp.1e16.Swan,K.,Garrison,D.R.,Richardson,J.,2009.Aconstructivistapproachtoonlinelearning:thecommunityofinquiryframework.In:Payne,C.R.(Ed.),Informa-tionTechnologyandConstructivisminHigherEducation:ProgressiveLearningFrameworks.IGIGlobal,Hershey,pp.43e57.Szabo,Z.,Schwartz,J.,2011.Learningmethodsforteachereducation:theuseofonlinediscussionstoimprovecriticalthinking.Technol.PedagogyEduc.20(1),79e94.http://dx.doi.org/10.1080/1475939X.2010.534866.Tappenden,S.,2011.Blendedandmobilelearning:experiencesfromaNewZealandfacultyoflaw.In:Kitchenham,A.(Ed.),BlendedLearningacrossDisciplines:ModelsforImplementation.InformationScienceReference,Hershey,pp.99e111.TheAmericanPhilosophicalAssociationDelphiReport,1990.CriticalThinking:aStatementofExpertConsensusforPurposesofEducationalAssessmentandInstruction.RetrievedDecember24,2015from.http://files.eric.ed.gov.Thomas,T.,2011.Developingfirstyearstudents’criticalthinkingskills.AsianSoc.Sci.7(4),26e34.Retrievedfrom.www.ccsenet.org/ass.Thormann,J.,Gable,S.,Fidalgo,P.S.,Blakeslee,G.,2013.Interaction,criticalthinking,andsocialnetworkanalysis(SNA)inonlinecourse.Int.Rev.Res.OpenDistanceLearn.14(3),294e317.Retrievedfrom.http://www.irrodl.org.Tofade,T.,Eisner,J.,Haines,S.T.,2013.Bestpracticestrategiesforeffectiveusofquestionsasateachingtool.Am.J.Pharm.Educ.77.7.Sept2013.http://dx.doi.org/10.5688/ajpe777155.Tolu,A.T.,Evans,L.,2013.Fromdistanceeducationtocommunitiesofinquiry:areviewofhistoricaldevelopments.In:Akyol,Z.,Garrison,D.R.(Eds.),Educa-tionalCommunitiesofInquiry:TheoreticalFramework,Research,andPractice.InformationScienceReference,Hershey,pp.45e66.Trees,A.R.,Jackson,M.H.,2007.Thelearningenvironmentinclickerclassrooms:studentprocessesoflearningandinvolvementinlargeuniversity-levelcoursesusingstudentresponsesystems.Learn,MediaTechnol.32(1),21e40.http://dx.doi.org/10.1080/17439880601141179.Wachira,P.,Keengwe,J.,2011.Technologyintegrationbarriers:urbanschoolmathematicsteachersperspectives.J.Sci.Educ.Technol.20,17e25.http://dx.doi.org/10.1007/s10956-010-9230-y.Whiteley,T.R.,2006.UsingtheSocraticmethodandBloom’sTaxonomyofthecognitivedomaintoenhanceonlinediscussion,criticalthinking,andstudentlearning.Dev.Bus.Simul.Exp.Learn.33,65e70.Retrievedfrom.https://journals.tdl.org.R.Swart/NurseEducationinPractice23(2017)30e3938

Yang,Y.-T.C.,Chuang,Y.-C.,Li,L.-Y.,Tseng,S.-S.,2013.Ablendedlearningenviron-mentforindividualizedEnglishlisteningandspeakingintegratingcriticalthinking.Comput.Educ.63,285e306.http://dx.doi.org/10.1016/j.compedu.2012.12.012.Yeh,Y.-C.,2012.Aco-creationblendedKMmodelforcultivatingcriticalthinkingskills.Comput.Educ.59(2012),1317e1327.http://dx.doi.org/10.1016/j.compedu.2012.05.017.Zhang,W.,Creswell,J.,2013.Theuseof“mixing”proceduresofmixedmethodsinhealthservicesresearch.Med.Care51(8)e51ee5.http://dx.doi.org/10.1097/mlr.0b013e31824642fd.R.Swart/NurseEducationinPractice23(2017)30e3939

ISSN 1477-7029 95 ©Academic Conferences Ltd Reference this paper as: Molina Azorín,J, M and Cameron, R. “The Application of Mixed Methods in Organisational Research: A Literature Review” The Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (pp.95-105), available online at www.ejbrm.com The Application of Mixed Methods in Organisational Research: A Literature Review José Molina Azorín1 and Roslyn Cameron2 1University of Alicante, Spain 2Southern Cross University, Tweed Heads, Australia [email protected] [email protected] Abstract: Mixed methods research (the combined use of quantitative and qualitative methods in the same study) is becoming an increasingly popular approach in the discipline fields of sociology, psychology, education and health sciences. Calls for the integration of quantitative and qualitative research methods have been advanced in these fields. A key feature of mixed methods research is its methodological pluralism, which frequently results in research which provides broader perspectives than those offered by monomethod designs. The overall purpose and central premise of mixed methods is that the use of quantitative and qualitative approaches in combination provides a better understanding of research problems and complex phenomena than either approach alone. Despite calls for the combined use of quantitative and qualitative research in business and management studies, the use of mixed methods in business and management has seldom been studied. The purpose of this paper is to review the application of mixed methods research within organisational research. The study reported in this paper identifies the use of mixed methods in three organisational journals for the period 2003 to 2009: the Strategic Management Journal, Journal of Organizational Behavior and Organizational Research Methods. The landmark Tashakkori and Teddlie (2003) Handbook of mixed methods in social and behavioral research, played a pivotal role in providing both the visibility and credibility of mixed methods as a third methodological movement and since the publication of this seminal work the mixed methods movement has rapidly gained popularity. Business and management researchers need to be made aware of the growing use and acceptance of mixed methods research across business and organisational journals. This paper examines the main characteristics of mixed methods studies identified in the sample in terms of purposes and designs, and posits suggestions on the application of mixed methodologies. Keywords: mixed methods research, strategic management, organizational behaviour, quantitative methods, qualitative methods 1. Introduction Mixed methods research (the combined use of quantitative and qualitative methods in the same study) is becoming an increasingly popular approach in the discipline fields of sociology, psychology, education and health sciences. Calls for the integration of quantitative and qualitative research methods have been advanced in these fields (Greene, Caracelli and Graham, 1989; O‟Cathain, 2009; Tashakkori and Teddlie 2003). The overall purpose and central premise of mixed methods is that the use of quantitative and qualitative approaches in combination may provide a better understanding of research problems and complex phenomena than either approach alone (Creswell and Plano Clark, 2007). Despite calls for the combined use of quantitative and qualitative research in management and organisational studies (Currall and Towler, 2003), the use of mixed methods in business and management has seldom been studied. The present study focuses on two main areas that represent micro and macro management research domains: organizational behavior and organizational strategy. These areas are the two largest divisions within the Academy of Management. Regarding research methods, scholars in both fields employ qualitative and quantitative methods, but the use of quantitative approaches dominate both strategy (Phelan, Ferreira and Salvador, 2002) and organizational behavior (Greenberg, 2007). Calls for methodological diversity and the use and integration of quantitative and qualitative research methods have been carried out in organizational behavior (Greenberg, 2007) and in strategy (Boyd, Gove and Hitt, 2005). Moreover, mixed methods research may play an important role not only in encouraging the use of a diversity of methods in these fields but also in bridging the micro and macro domains. The purpose of this paper is to review the application of mixed methods research within organisational and management research. The study reported in this paper identifies the use of mixed methods in the leading journals in these two organisational fields: the Strategic Management Journal (SMJ) and the Journal of Organizational Behavior (JOB). In addition, Organizational Research Methods (ORM) is

Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (95 – 105) www.ejbrm.com 96 ©Academic Conferences Ltd also examined. Management and organisational researchers need to be made aware of the growing use and acceptance of mixed methods research within these fields. This paper examines the main characteristics of mixed methods studies (purposes and designs) and posits suggestions on the application of mixed methodologies. The paper is organised as follows. First, several important aspects of mixed methods research are examined (definitions, applications, barriers and benefits). Subsequently, the methodology employed, including the sample selection and search strategies for identifying the mixed methods studies, is described. This is followed by a discussion of the main characteristics of the articles that have been identified as mixed methods studies. The paper concludes with suggestions on the application of mixed methods research in management and possible future research directions. 2. Mixed methods research 2.1 Definition A monomethod study uses only one type of method, one quantitative or one qualitative. In general, in a quantitative study, the data is in numerical form and this information is analyzed using quantitative data analysis techniques. In a qualitative study, the information, which is mainly in textual form, is analyzed employing qualitative data analysis techniques. Drawing an initial distinction between monomethod research and multiple methods research may be helpful to determine what is understood as „mixed methods‟. A multiple methods study uses more than one method. Moreover, a differentiation can be made within multiple method designs between multimethod research (multiple qualitative or quantitative methods) and mixed methods research (integration of quantitative and qualitative methods) (Creswell and Plano Clark, 2007). Several definitions exist for mixed methods research. Greene, Caracelli and Graham (1989) defined mixed methods research designs as those that include at least one quantitative method (designed to collect numbers) and one qualitative method (designed to collect words). Tashakkori and Teddlie (1998) refer to mixed methods studies as those that combine the qualitative and quantitative approaches into the research methodology of a single study. Johnson and Onwuegbuzie (2004) indicated that mixed methods research is the class of research where the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study. In this paper, the following definition supplied by Plano Clark (2005) is applied in this study: mixed methods research is research that combines qualitative and quantitative data collection and data analysis within a single study. 2.2 Is the application of mixed methods research possible? Several debates or “wars” (Datta, 1994) have raged in the social sciences regarding the superiority of one or the other of the two major social science paradigms: the positivist approach and the constructivist orientation. Along with superiority, another important aspect examined in those debates has been the idea of incommensurability and incompatibility, which means that qualitative and quantitative approaches could/should not be used in the context of the same study. This idea is based on the fact that there are quite different epistemological and ontological assumptions that underpin different paradigms and methods. The organising idea of a continuum, with hard positivism at one end and constructionism at the other, can be used to point out the main epistemological and ontological assumptions. A hard positivism ontology asserts that an objective reality is out there to be found and epistemologically this can be done with knowable degrees of certainty using objectively-correct scientific methods (Long, White, Friedman and Brazeal, 2000). The result is certain knowledge, and concepts such as reliability, validity and statistical significance are used carefully in good hard-positivist research with the purpose of describing some part of reality with certainty. A softer version of positivism also infers that objective reality exists, but epistemologically suggests that techniques to uncover the world produce probabilistic and ultimately uncertain understandings. Constructionism has a relativist ontology, that is, each person has his or her own reality. Epistemologically, the achievement of objectivity is rejected, and emphasis is placed on individual understanding of particular viewpoints. The positivist paradigm underlies what are called quantitative methods, while the constructivist paradigm is more related to qualitative methods (Howe, 1988). Therefore, the debate between these paradigms has also been called the qualitative-quantitative debate (Reichardt and Rallis, 1994). In

José Molina-Azorín and Roslyn Cameron www.ejbrm.com 97 ISSN 1477-7029 this debate, many purist researchers were very stringent not only in their defense of their own methodological positions but also in their attack on the position of the “other side”. Both sets of purists view their paradigms as the ideal for research, and, implicitly if not explicitly, they advocate the incompatibility thesis (Howe, 1988), which posits that qualitative and quantitative research paradigms cannot and should not be mixed, and then compatibility between quantitative and qualitative methods is impossible due to the incompatibility of the paradigms underlying the methods. This led to a dichotomy between these two paradigms and between these two general groups of methods. From this paradigmatic viewpoint, to accept the combination and mix of different approaches and methods is misleading as these methodologies are derived from fundamentally different epistemological positions and are therefore incommensurable. However, there have been numerous calls and attempts in the social sciences to make peace between the two major paradigmatic positions. “Pacifists” have appeared who state that qualitative and quantitative methods are compatible. Reichardt and Cook (1979) countered the incompatibility thesis based on the paradigm-method fit by suggesting that different philosophical paradigms and methods are compatible. Moreover, these authors argued that paradigms and methods are not inherently linked. Mir and Watson (2000) pointed out that a researcher who is anchored in constructivist methodology may employ a variety of methods including statistical analysis, just as a researcher employing a realist methodology may use qualitative research. Thus, research methods are more independent of epistemological and ontological assumptions than is sometimes supposed (Bryman and Bell, 2007). In addition, some researchers try to find philosophical assumptions that form the foundation for conducting mixed methods research. From this point of view, it can be pointed out two positions: the use of multiple worldviews and the use of a single paradigm or worldview (Creswell and Creswell, 2005). Regarding the first position, Greene and Caracelli (1997) advance a “dialectical” perspective in which researchers use multiple worldviews. This perspective maintains that mixed methods research may be viewed strictly as a “method”, thus allowing researchers to use any number of philosophical foundations for its justification and use. The second position is related to the best paradigm issue. The key question is what philosophical paradigm is the best foundation for mixed methods research. Many authors advocate for a pragmatism worldview for mixed methods (Cherryholmes, 1992). Pragmatism advances multiple pluralistic approaches to knowing, using “what works”, a focus on the research questions as important with all types of methods to follow to answer the questions, and a rejection of a forced choice between postpositivism and constructivism. Thus, a major tenet of pragmatism is that quantitative and qualitative methods are compatible. 2.3 Barriers and benefits of mixed methods research Mixed methods research is not intrinsically superior to research that relies on a single method. An important consideration prior to designing and conducting a mixed methods study is whether mixed methods, as compared to monomethod designs, best addresses the research problem and the research question(s). Moreover, there are important barriers to conducting a mixed methods study. Creswell and Plano Clark (2007) pointed out that conducting mixed methods research is not easy. Mixed methods studies are a challenge because they are perceived as requiring more work and financial resources, and they take more time. Increased time demands arise from the time it takes to implement both aspects of the study (Niglas, 2004). In addition, mixed methods research also requires that researchers develop a broader set of skills that span both the quantitative and the qualitative. Another barrier is related to the challenges of publishing mixed methods studies (Plano Clark, 2005; Bryman, 2007). The challenges generally arise from existing constraints such as word and page limits in journals. In spite of these barriers, mixed methods research is carried out and published. Several authors have examined the utility and advantages of this approach. Regarding the main benefits, the overall purpose and central premise of mixed methods studies is that the use of quantitative and qualitative approaches in combination may provide a better understanding of research problems and complex phenomena than either approach alone (Creswell and Plano Clark, 2007). Better understanding can be obtained, for example, by triangulating one set of results with another and thereby enhancing the validity of inferences. In fact, the concept of triangulation of methods was the intellectual wedge that eventually broke the methodological hegemony of the monomethod purists (Tashakkori and Teddlie, 1998). Jick (1979) discussed triangulation in terms of the weaknesses of one method being offset by the strengths of another. It is often stressed that different methods have different weaknesses and strengths, and therefore the main effect that triangulation can offer is to overcome the weaknesses of

Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (95 – 105) www.ejbrm.com 98 ©Academic Conferences Ltd any single method. Thus, if we use several different methods for investigating the phenomenon of our interest, and the results provide mutual confirmation, we can be more confident that our results are valid (Niglas, 2004). Other purposes, reasons or rationales for combining qualitative and quantitative methods can also be pointed out. These purposes may be considered as benefits and advantages of mixed methods research. Greene, Caracelli and Graham (1989) stated four additional purposes along with triangulation: complementarity (seeking elaboration, illustration, enhancement, and clarification of the results from one method with the findings from the other method), development (when the researcher uses the results from one method to help develop or inform the use of the other method), initiation (discovering paradoxes and contradictions that lead to the research questions being reframed), and expansion (seeking to extend the breadth and range of inquiry by using different methods for different inquiry components). With respect to these main purposes, other authors indicated a wider range of reasons. For example, Collins, Onwuegbuzie and Sutton (2006) provided a comprehensive list of reasons or purposes for conducting mixed methods research, and each of these purposes was grouped under one of the four main rationales: participant enrichment, instrument fidelity, treatment integrity, and significance enhancement. Bryman and Bell (2007) also presented a variety of purposes in mixed methods research: triangulation, qualitative research facilitates quantitative research, quantitative research facilitates qualitative research, analysis of static and processual features, qualitative research may facilitate the interpretation of the relationship between the variables, and analysis of different aspects of a phenomenon. 2.4 The application of mixed methods research Two main factors that help researchers to design and conduct a mixed methods study are implementation of data collection and priority (Morse 1991; Morgan 1998; Tashakkori and Teddlie 1998; Creswell 2003). Implementation of data collection refers to the sequence that the researcher uses to collect both quantitative and qualitative data. The options consist of gathering the information at the same time (concurrent, simultaneous or parallel design) or introducing the information in phases (sequential or two-phase design). By concurrently gathering both the forms of data, the researcher seeks to compare them with the search for congruent findings. When the data are introduced in phases, either the qualitative or the quantitative approach may be gathered first, but the sequence relates to the objectives being sought by the researcher. Thus, when qualitative data collection precedes the quantitative data collection, the intent is to first explore the problem under study and then follow up on this exploration with quantitative data that are amenable to studying a large sample so that results might be inferred to a population. Alternatively, when quantitative data precede the qualitative data, the intent is to test the variables with a large sample and then carry out a more in-depth exploration of a few cases during the qualitative phase. Regarding priority, mixed methods researchers can give equal priority to both quantitative and qualitative research, or emphasize more on qualitative or quantitative parts. This emphasis may result from research questions, practical constraints for data collection, the need to understand one form of data before proceeding to the next or the audience preference. Mixed methods designs can therefore be divided into equivalent status designs (the researcher conducts the study using both the quantitative and qualitative approaches equally to understand the phenomenon under study) and dominant-less dominant studies or nested designs (the researcher conducts the study within a single dominant paradigm with a small component of the overall study drawn from an alternative design). These two dimensions and their possible combinations can lead to the establishment of several designs which are represented using the notation proposed by Morse (1991). In her system, the main or dominant method appears in capital letters (QUAN, QUAL) whereas the complementary method is given in lowercase letters (quan, qual). The notation “+” is used to indicate a simultaneous design, and the arrow “→” stands for sequential design. Thus, the following four groups and nine types of mixed methods designs can exist using these two dimensions (Johnson and Onwuegbuzie 2004):  I- Equivalent status/simultaneous design: QUAL+QUAN.  II- Equivalent status/sequential designs: QUAL→QUAN; QUAN→QUAL.  III- Dominant/simultaneous designs: QUAL+quan; QUAN+qual.

José Molina-Azorín and Roslyn Cameron www.ejbrm.com 99 ISSN 1477-7029  IV- Dominant/sequential designs: qual→QUAN; QUAL→quan; quan→QUAL; QUAN→qual. 3. Methods A review of the use of mixed methods in the SMJ, JOB and ORM over a seven year period (2003-2009) was carried out. The beginning of this time period aligns with the publication of two very important works signifying the growing emergence of mixed methods as a methodological movement in its own right. In 2003 Tashakkori and Teddlie‟s (2003) landmark Handbook of mixed methods in social and behavioral research was published, which played a pivotal role in providing both visibility and credibility to the field of mixed methods. In the same year, Creswell‟s seminal book on research design across the three approaches (qualitative, quantitative and mixed methods) was also published (Creswell, 2003). This time period is recent, giving us the opportunity to see how research designs are applied in current settings. Moreover, the time period spans seven years, providing a long enough interval representing research within these fields. An important aspect related to identification and prevalence of mixed methods articles is the search strategy used. In this study, we have utilised two main search strategies: an electronic online word search and a manual search. The rationale behind this decision was to investigate the use of mixed methods terminology and nomenclature within the empirical studies employing a mixture of both qualitative and quantitative methods in a single study. Teddlie and Tashakkori (2003, p. 8) state that „the development of a nomenclature that is distinctly associated with mixed methods is both extremely important and overdue‟. The online searches were conducted through the SMJ and JOB website (Wiley InterScience database) and the ORM journal website (SAGE publishing). This electronic search strategy utilised the following phrases: “mixed methods”, “mixed method”, “multi-methods” and “multi-method”. The searches were filtered for the following sections of the articles: title; abstract; keywords and; full text for ORM and for the SMJ and JOB the searches were through the following sections: title; keywords and; abstract/full text. Each article identified in the searches was checked to determine what context the search words were being utilised for (e.g., the application of a mixed methods research design; book review; description of other studies being discussed, methodological discussion). In addition, a manual search was used for the SMJ and JOB. Thus, all articles published in SMJ (n=498) and JOB (n=373) from 2003 to 2009 were read and reviewed. Bryman (2006) points out that the electronic search strategy may provide a biased sample of mixed methods studies in the sense that by no means all authors of articles reporting mixed methods research foreground the fact that the findings reported derive from a combination of quantitative and qualitative research, or do not do so in terms of the key words that drove the online search strategy. Moreover, apart from identifying mixed methods studies, the manual search strategy can be used to classify the articles in two main groups, non-empirical and empirical articles, and, additionally, the group of empirical studies can be further divided into three types: quantitative, qualitative and mixed methods articles. A sequential mixed methods study with two stages was undertaken to identify mixed methods articles and determine their main characteristics. In the first phase, a qualitative stage was used in the manual search strategy for the purpose of determining whether each article represented a non-empirical, quantitative, qualitative or mixed methods study. This content analysis involved using all information presented in each article (title, abstract, keywords, introduction, literature review, methods, results, discussion and conclusions). This initial data collection created individual tables by journal and by year, listing the journal title, year, volume and issue number, number of total articles, number of non-empirical articles, number of quantitative articles, number of qualitative articles and number of mixed methods articles. These tables were aggregated across journals. In addition, the mixed methods articles identified were re-examined through a content analysis and coded according to two main dimensions: priority (equal or dominant status of the quantitative and qualitative parts) and implementation of data collection (simultaneous or sequential). The main mixed methods purposes were also determined. In the second phase, descriptive statistics were used for the quantitative analysis for prevalence, providing numbers, sums and percentages by type of article (non-empirical, quantitative, qualitative and mixed methods articles) and by journal. In addition, regarding the mixed methods articles, numbers and percentages by type of priority (equal or dominant status), type of implementation (simultaneous or sequential) and purpose were also provided.

Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (95 – 105) www.ejbrm.com 100 ©Academic Conferences Ltd 4. Results 4.1 Strategic Management Journal The electronic search utilising the “mixed method(s)” and “multi-method(s)” phrases identified 10 articles in the SMJ for 2003-2009. Upon closer examination, 9 of the SMJ identified articles were classified as QUAN studies and 1 was conceptual. Therefore, this electronic search strategy did not identify any empirical mixed methods studies. Through the manual search, the 498 articles published were read. Table 1 shows the prevalence frequencies of each type of article by year. Table 1: Types of articles in the Strategic Management Journal (2003-2009) Year Total number of articles Number of non-empirical articles Empirical articles Total number of empirical articles Number of quantitative articles Number of qualitative articles Number of mixed articles 2003 78 12 66 54 6 6 2004 67 4 63 51 2 10 2005 70 9 61 47 0 14 2006 63 7 56 43 2 11 2007 73 9 64 53 5 6 2008 76 6 70 64 1 5 2009 71 6 65 53 0 12 Total 498 53 445 365 16 64 As can be seen in Table 1, the SMJ is dominated by quantitative articles (365 articles, 73.3%) The second type of article in importance is non-empirical articles (53 articles, 10.6%), and the number of mixed methods studies (64 articles, 12.9%) is higher than the number of qualitative articles (16 studies, 3.2%). One way to organise and describe the 64 mixed methods articles published in the SMJ is to examine the designs and the purposes of these studies. Through a content analysis, identification of the main characteristics of the mixed methods articles published in the SMJ has been carried out (see Table 2). The most common purpose was development (82.8%), followed by complementarity (9.4%), expansion (4.7%) and triangulation (3.1%). Regarding implementation of data collection, sequential implementation was the most common implementation pattern used in the mixed methods articles published in the SMJ (61 articles, 95.3%). In these sequential articles, the first part in most studies was the qualitative one (54 studies). Finally, the most common type of priority was different priority of the quantitative and qualitative parts (53 articles, 82.8%). Moreover, in these 53 articles, the dominant part was the quantitative one. Table 2: Characteristics of mixed methods studies in the Strategic Management Journal (2003-2009) Purposes N (%) Triangulation 2 (3.1%) Complementarity 6 (9.4%) Development 53 (82.8%) Expansion 3 (4.7%) Implementation Simultaneous 3 (4.7%) Sequential 61 (95.3%) Priority Equivalent status 11 (17.2%) Dominant 53 (82.8%) Therefore, the typical mixed methods article in the SMJ is a qual→QUAN study where the main mixed methods purpose is development (see for example Ethiraj, Kale, Krishnan and Singh, 2005). Several aspects of the quantitative part may be improved through the use of a previous qualitative phase. Thus, the qualitative part may help to develop or extend theory, identify the industry-specific

José Molina-Azorín and Roslyn Cameron www.ejbrm.com 101 ISSN 1477-7029 resources and competences as well as the dependent variables and/or improve the measurement instrument of the quantitative phase. 4.2 Journal of Organizational Behavior In this case, the online search utilising the “mixed method(s)” and “multi-method(s)” phrases identified 13 articles in the JOB for 2003-2009. However, upon closer examination, 10 of the 13 articles were quantitative studies, 2 conceptual and 1 was considered to be a mixed methods study (Challiol and Mignonac, 2005). Using the manual search which involved reading all the articles published in the JOB from 2003 to 2009 (373 articles), 20 mixed methods studies were identified (5.4%). Table 3 shows the number of articles classified by article type and year. As is the case with the SMJ, the JOB is also dominated by quantitative articles (235 articles, 63%) followed by non-empirical articles (101 articles, 27.1%). Moreover, the number of mixed methods studies is also higher than the number of qualitative articles (17 studies, 4.5%). Table 3: Types of articles in the Journal of Organizational Behaviour (2003-2009) Year Total number of articles Number of non-empirical articles Empirical articles Total number of empirical articles Number of quantitative articles Number of qualitative articles Number of mixed articles 2003 50 10 40 40 0 0 2004 47 4 43 35 2 6 2005 50 18 32 28 1 3 2006 56 15 41 33 5 3 2007 55 20 35 27 5 3 2008 60 20 40 34 4 2 2009 55 14 41 38 0 3 Total 373 101 272 235 17 20 In order to determine the characteristics of the 20 mixed methods studies, a content analysis was undertaken. Table 4 shows these characteristics with regard to purposes, implementation and priority. In the JOB, the most common purpose was development (45%), followed by complementarity (35%), triangulation (15%) and expansion (5%). Sequential implementation was the most common implementation pattern used in the mixed methods articles identified (15 articles, 75%). In these 15 sequential articles, the first part in most studies was the qualitative one (9 studies). Finally, the most common type of priority was different priority of the quantitative and qualitative parts (12 articles, 60%). Moreover, in these 12 articles, the dominant part was the quantitative one. Table 4: Characteristics of mixed methods studies in Journal of Organizational Behavior (2003-2009) Purposes N (%) Triangulation 3 (15%) Complementarity 7 (35%) Development 9 (45%) Expansion 1 (5%) Implementation Simultaneous 5 (25%) Sequential 15 (75%) Priority Equivalent status 8 (40%) Dominant 12 (60%) 4.3 Organizational Research Methods As stated earlier, in the case of ORM we conducted the electronic search and not the manual search. The online search identified 8 documents, which includes a book reviews and a methodological paper that made mention of mixed methods as a methodology within their respective discussions. Only two articles may be considered as mixed methods empirical studies: Yauch and Steudel (2003) and Pratt (2008).

Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (95 – 105) www.ejbrm.com 102 ©Academic Conferences Ltd For example, Yauch and Steudel (2003) is an exemplar of QUAL+quan design. These authors analyzed an important intangible resource, specifically the organizational cultures of two small manufacturers using qualitative and quantitative data. The article described not only how qualitative and quantitative data contributed to the validity of the results through triangulation but also how the qualitative and quantitative research paradigms were used in a complementary fashion to produce a more complete understanding of the organizational cultures. Using methods from both research paradigms enabled a greater understanding of cultural artefacts and behaviors but more important of the underlying cultural values and assumptions. A prospective exploratory case study approach was used to examine the impact of organizational culture on the cellular manufacturing conversion process. The ultimate goal of the research was to identify key cultural factors that had a positive or negative impact on the process of converting form a traditional functional manufacturing system to cellular manufacturing. Cultural assessment of the organizational values, assumptions, and behavioral norms was accomplished through qualitative and quantitative means. Qualitative assessment of culture was accomplished through document review, participant observation and group interviews. The Organizational Culture Inventory, a cultural assessment survey, was used as an additional measure of organizational culture at each company. This study relied most heavily on qualitative data but supplemented it with quantitative survey results. The different results from the two search strategies brings to light not only the use of mixed methods research in the sample but a strong indication that commonly used mixed methods terminology and nomenclature emerging from the mixed methods movement is not being utilised in these studies. 5. Discussion and conclusions Creswell and Tashakkori (2007) pointed out that the literature base about mixed methods research may not be well known to individuals in specific fields. To check this issue, the references sections of the mixed methods articles identified were reviewed. Despite the availability of mixed-methods-related books, book chapters and journal articles, studies about this type of research have seldom been found in the references sections of these mixed methods studies. On this evidence it seems likely that the advantages, possibilities, purposes, designs and potential of mixed methods research may be unknown to researchers in these fields. Put differently, mixed methods research is used in organisational and management studies but it may be completely unknown and without recognition that mixed methods research constitutes a specific approach to research. Therefore, although mixed methods research is used in these business fields, these mixed methods studies may not exploit the full potential for mixing methods and researchers are probably not maximizing the extent to which they are using this approach. Some authors provide guidelines about how to carry out a mixed methods research. Creswell (1999) offers nine steps in conducting a mixed methods study:  Determine if a mixed methods study is needed to study the problem.  Consider whether a mixed methods study is feasible.  Write both qualitative and quantitative research questions.  Review and decide on the types of data collection.  Assess the relative weight and implementation strategy for each method.  Present a visual model.  Determine how the data will be analysed.  Assess the criteria for evaluating the study.  Develop a plan for the study. Similarly, Teddlie and Tashakkori (2006) provide a seven step process for researchers selecting the best design for their projects:  The researcher must first determine if his/her research question require a monomethod or mixed methods design.  The researcher should be aware that are a number of different typologies of mixed methods research designs and should know how to access details regarding them.

José Molina-Azorín and Roslyn Cameron www.ejbrm.com 103 ISSN 1477-7029  The researcher wants to select the best mixed methods design for his/her particular study and assumes that one of the published typologies includes the right design for the project.  Typologies may be differentiated by the criteria that are used to distinguish among the research designs within them, and the researcher needs to know those criteria.  These criteria should be listed by the researcher, who may then select the criteria that are most important for the particular study he/she is designing.  The researcher then applies the selected criteria to potential designs, ultimately selecting the best research design.  In some cases, the researcher may have to develop a new mixed methods design, because no one best design exists for his/her research project. Regarding the last step, Teddlie and Tashakkori (2006) point out that mixed methods designs have an opportunistic nature. Thus, in many cases, a mixed methods research study may have a predetermined research design, but new components of the design may evolve as researchers follow up on leads that develop as data are collected and analysed. The point is for the researcher to be creative and not be limited by the existing designs. Hanson, Creswell, Plano Clark, Petska and Creswell (2005) offer some recommendations for designing, implementing, and reporting a mixed methods study. Thus, they recommend that researchers attend closely to design and implementation issues, particularly to how and when data are collected (e.g., concurrently or sequentially). The study‟s purpose plays an important role here. They also recommend that researchers familiarize themselves with the analysis and integration strategies used in the published mixed methods studies. Moreover, because mixed methods studies require a working knowledge and understanding of both quantitative and qualitative methods, and because they involve multiple stages of data collection and analysis that frequently extend over long periods of time, they recommend that researchers work in teams. Moreover, in preparing a mixed methods manuscript, they recommend that researchers use the phrase “mixed methods” in the titles of their studies, and that, early on, researchers foreshadow the logic and progression of their studies by stating the study‟s purpose and research questions in the introduction. Clear, well written purpose statements and research questions that specify the quantitative and qualitative aspects of the study help focus the manuscript. Additionally, these authors recommend that, in the introduction, researchers explicitly state a rationale for mixing quantitative and qualitative methods and data (e.g., to triangulate results, to develop or improve one method with the other, to extend the study‟s results). Another recommendation is that, in the methods, researchers specify the type of mixed methods research design used. Creswell and Plano Clark (2010) provide several principles for designing a mixed methods study:  Recognize that mixed methods designs can be fixed and/or emergent. Fixed mixed methods designs are mixed methods studies where the use of quantitative and qualitative methods is predetermined and planned at the start of the research process, and the procedures are implemented as planned. Emergent mixed methods designs are found in mixed methods studies where the use of mixed methods arises due to issues that develop during the process of conducting the research.  Identify an approach to design. There are several approaches to design, and researchers can benefit from considering their personal approach to conducting mixed methods studies. These design approaches fall into two categories: typology-based and dynamic.  Match the design to the research problem, purpose and questions. The importance of the research problem and questions is a key principle of mixed methods research design. This perspective stems from the pragmatic foundations for conducting mixed methods research where the notion of “what works” applies well to selecting the methods that work best to address a study‟s problem and questions.  Be explicit about the reasons for mixing methods. Another key principle of mixed methods design is to identify the reason(s) for mixing quantitative and qualitative methods within the study. Combining methods is challenging and should only be undertaken when there is a specific reason to do so. Two frameworks have been provided by Greene, Caracelli and Graham (1989) and Bryman (2006). In addition, Creswell and Plano Clark (2010) also point out several key decisions in choosing a mixed methods design:

Electronic Journal of Business Research Methods Volume 8 Issue 2 2010 (95 – 105) www.ejbrm.com 104 ©Academic Conferences Ltd  Determine the level of interaction between the quantitative and qualitative strands.  Determine the priority of the quantitative and qualitative strands.  Determine the timing of the quantitative and qualitative strands.  Determine where and how to mix the quantitative and qualitative strands. Regarding the conclusions derived from this study, several aspects may be emphasised. First, with regard to the characteristics of the mixed methods articles identified, the most common purpose in strategy and organizational behaviour was development; the most common type of priority was different priority of the quantitative and qualitative parts; and sequential implementation of data collection was the most common implementation pattern used in these fields. Second, the number of mixed methods studies shows great variation year to year and journal to journal, and there is not a clear trend in the publication of mixed methods research in business studies for the journals studied. Third, with regard to the search strategies to identify mixed methods studies, the online search and the manual search provided very different results. As said above, Bryman (2006) pointed out that the electronic search strategy may provide a biased sample of mixed methods studies. It may also indicate the lack of awareness within the management research community of the emerging body of literature and methodological frameworks being developed from within the mixed methods movement. In any case, this electronic search may be further expanded to use a greater range of mixed methods terminology and nomenclature. We would like to indicate that we agree with the “paradigm of choices” emphasized by Patton (1990). A paradigm of choices rejects methodological orthodoxy in favour of methodological appropriateness as the primary criterion for judging methodological quality. Thus, this paradigm of choices recognizes that different methods are appropriate for different situations. The predominance of more quantitative-based methodological tools in the development of strategy and organisational behaviour research does not mean that these tools are applicable to all research questions. The research question and context should dictate the choice of the appropriate research methods. However, we must also take into account that the knowledge about mixed methods research can stimulate a researcher to better define and analyze innovative problems and research questions in management research. Mixing methods therefore offers enormous potential for exploring new dimensions (Mason, 2006). Hopefully, this review of management empirical studies which have used mixed methods designs along with the ideas offered for the application of mixed methods studies may favour progress on management research. Finally, with regard to future research, although the current study attempts to extend the knowledge of the application of mixed methods research in management research, much remains to be learned. For example, it would be interesting to analyze the yield from mixed methods studies regarding the added value of these articles, or the contribution to the improvement of several methodological aspects such as validity or construct measurement. Moreover, an analysis of the use and application of mixed methods research in other organisational and management fields would also be interesting and could expand upon the research reported here. References Boyd, B., Gove, S. and Hitt, M. (2005) “Construct Measurement in Strategic Management Research: Illusion or Reality?”, Strategic Management Journal, Vol. 26, pp 239-257. Bryman, A. (2006) “Integrating Quantitative and Qualitative Research: How is it Done?”, Qualitative Research, Vol. 6, pp 97-113. Bryman, A. (2007). “Barriers to Integrating Quantitative and Qualitative Research”, Journal of Mixed Methods Research, Vol. 1, pp 8-22. Bryman, A. and Bell, E. (2007) Business Research Methods (2nd Ed.), Oxford University Press, Oxford. Challiol, H. and Mignonac K. (2005) “Relocation Decision-making and Couple Relationships: A Quantitative and Qualitative Study of Dual-earner Couples”, Journal of Organizational Behavior, Vol. 26, pp 247-274. Cherryholmes, C. (1992) “Notes on Pragmatism and Scientific Realism”, Educational Researcher, Vol. 21, pp. 13-17. Collins, K.M.T., Onwuegbuzie, A.J., and Sutton, I.L. (2006) “A Model Incorporating the Rationale and Purpose for Conducting Mixed-Methods Research in Special Education and Beyond”, Learning Disabilities. A Contemporary Journal, Vol. 4, pp 67-100. Creswell, J. (1999) Mixed-Method Research: Introduction and Application. In Handbook of Educational Policy, Cizek, G. (ed.), Academic Press, San Diego, pp 455-472. Creswell, J. (2003) Research Design. Qualitative, Quantitative and Mixed Methods Approaches (2nd ed.), Sage, Thousand Oaks.

José Molina-Azorín and Roslyn Cameron www.ejbrm.com 105 ISSN 1477-7029 Creswell, J. and Creswell, J. (2005) “Mixed Methods Research: Developments, Debates, and Dilemmas”. In Research in Organizations. Foundations and Methods of Inquiry, Swanson, R. and Holton III, E. (eds.), Berrett-Koehler Publishers, San Francisco, pp. 315-326. Creswell, J. and Plano Clark, V. (2007) Designing and Conducting Mixed Methods Research, Sage, Thousand Oaks. Creswell, J. and Plano Clark, V. (2010) Designing and Conducting Mixed Methods Research, 2nd Edition, Sage, Thousand Oaks. Currall, S. and Towler A. (2003) “Research Methods in Management and Organizational Research: Toward Integration of Qualitative and Quantitative Techniques”. In Handbook of Mixed Methods in Social & Behavioral Research, Tashakkori, A. and Teddlie, C. (eds.), Sage, Thousand Oaks, pp 513-526. Datta, L. (1994) “Paradigm Wars: A Basis for Peaceful Coexistence and Beyond”. In The Qualitative-Quantitative Debate: New Perspectives, Reichardt, C. and S. Rallis (eds.), Jossey-Bass, San Francisco, pp. 53-70. Ethiraj, S., Kale, P., Krishnan, M. and Singh, J. (2005) “Where do Capabilities Come From and How do They Matter? A study in the Software Services Industry”, Strategic Management Journal, Vol. 26, pp 25-45. Greenberg, J. (2007) “A Plea for Methodological diversity”, Journal of Organizational Behavior, Vol. 28, pp 929-931. Greene, J. and Caracelli, V. (eds.) (1997) Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms. Jossey-Bass, San Francisco. Greene, J., Caracelli, V. and Graham W. (1989) “Toward a Conceptual Framework for Mixed-Method Evaluation Designs”, Educational Evaluation and Policy Analysis, Vol. 11, pp 255-274. Hanson, W., Creswell, J., Plano Clark, V., Petska, K. and Creswell, J. (2005) “Mixed Methods Research Designs in Counseling Psychology”, Journal of Counseling Psychology, Vol. 52, pp 224-235. Howe, K. (1988) “Against the Quantitative-Qualitative Incompatibility Thesis (or Dogmas Die Hard)”, Educational Researcher, Vol. 17, No 8, pp. 10-16. Jick, T. (1979) “Mixing Qualitative and Quantitative Methods: Triangulation in Action”, Administrative Science Quarterly, Vol. 24, pp 602-611. Johnson, B. and Onwuegbuzie A. (2004) “Mixed Methods Research: A Research Paradigm Whose Time Has Come”, Educational Researcher, Vol. 33, No. 7, pp 14-26. Long, R., White, M., Friedman, W. and Brazeal, D. (2000) “The „Qualitative‟ versus „Quantitative‟ Research Debate: A Question of Metaphorical Assumptions?”, International Journal of Value-Based Management, Vol. 13, pp. 189-197. Mason, J. (2006) “Mixing Methods in a Qualitatively Driven Way”, Qualitative Research, Vol. 6, pp 9-25. Mir, R. and Watson, A. (2000) “Strategic Management and the Philosophy of Science: The case for Constructivist Methodology”, Strategic Management Journal, Vol. 21, pp. 941-953. Morgan, D. (1998) “Practical Strategies for Combining Qualitative and Quantitative Methods: Applications to Health Research”, Qualitative Health Research, Vol. 8, pp 362-376. Morse, J. (1991) “Approaches to Qualitative-Quantitative Methodological Triangulation”, Nursing Research, Vol. 40, pp 120-123. Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational Research, Tallinn Pedagogical University Press, Tallinn. O‟Cathain, A. (2009) “Mixed Methods Research in the Health Sciences. A Quiet Revolution”, Journal of Mixed Methods Research, Vol. 3, pp 3-6. Patton, M. (1990) Qualitative Evaluation and Research Methods (2nd ed.), Sage, Newbury Park. Phelan, S., Ferreira, M. and Salvador, R. (2002) “The first twenty years of the Strategic Management Journal”, Strategic Management Journal, Vol. 23, pp 1161-1168. Plano Clark, V.L. (2005) Cross-disciplinary Analysis of the Use of Mixed Methods in Physics Education Research, Counseling Psychology, and Primary Care. Doctoral dissertation, University of Nebraska-Lincoln. Pratt, M. (2008) “Fitting Oval Pegs into Round Holes. Tensions in Evaluating and Publishing Qualitative Research in Top-Tier North American Journals”, Organizational Research Methods, Vol. 11, pp 481-509. Reichardt, C. and Cook, T. (1979) “Beyond Qualitative versus Quantitative Methods. In Qualitative and Quantitative Methods in Evaluation Research, Cook, T. and Reichardt, C. (eds.), Sage, Thousand Oaks, pp. 7-32. Reichardt, C. and Rallis, S. (1994) The Qualitative-Quantitative Debate: New Perspectives, Jossey-Bass, San Francisco. Tashakkori, A. and Teddlie, C. (1998) Mixed Methodology. Combining Qualitative and Quantitative Approaches, Sage, Thousand Oaks. Tashakkori, A. and Teddlie, C. (Eds.) (2003) Handbook of Mixed Methods in Social & Behavioral Research, Sage, Thousand Oaks. Teddlie, C. and Tashakkori, A. (2003) “Major Issues and Controversies in the Use of Mixed Methods in the Social and Behavioural Sciences”. In Handbook of Mixed Methods in Social & Behavioral Research, Tashakkori, A. and Teddlie, C. (eds.), Sage, Thousand Oaks, pp 3-50. Teddlie, C. and Tashakkori, A. (2006) “A General Typology of Research Designs Featuring Mixed Methods”, Research in the Schools, Vol. 13, pp 12-28. Yauch, C. and Steudel, H. (2003) “Complementary Use of Qualitative and Quantitative Cultural Assessment Methods”, Organizational Research Methods, Vol. 6, pp 465-481.

Copyright of Electronic Journal of Business Research Methods is the property of Academic Conferences, Ltd.and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyrightholder’

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.