Mixed-Methods Research in Nutrition and Dietetics

Published:March 08, 2017DOI:https://doi.org/10.1016/j.jand.2017.01.018

      Abstract

      This work focuses on mixed-methods research (MMR) and is the 11th in a series exploring the importance of research design, statistical analysis, and epidemiologic methods as applied to nutrition and dietetics research. MMR research is an investigative technique that applies both quantitative and qualitative data. The purpose of this article is to define MMR; describe its history and nature; provide reasons for its use; describe and explain the six different MMR designs; describe sample selection; and provide guidance in data collection, analysis, and inference. MMR concepts are applied and integrated with nutrition-related scenarios in real-world research contexts and summary recommendations are provided.

      Keywords

      The Continuing Professional Education (CPE) quiz for this article is available for free to Academy members through the MyCDRGo app (available for iOS and Android devices) and via www.eatrightPRO.org. Simply log in with your Academy of Nutrition and Dietetics or Commission on Dietetic Registration username and password, go to the My Account section of My Academy Toolbar, click the “Access Quiz” link, click “Journal Article Quiz” on the next page, then click the “Additional Journal CPE quizzes” button to view a list of available quizzes. Non-members may take CPE quizzes by sending a request to [email protected] . There is a fee of $45 per quiz for non-member Journal CPE. CPE quizzes are valid for 1 year after the issue date in which the articles are published.
      The purpose of this work is to assist registered dietitian nutritionists (RDNs) and aspiring nutrition researchers apply and interpret mixed-methods research (MMR) principles consistent with high quality nutrition research. Most RDNs are familiar with the two major research design paradigms, quantitative (QUAN) and qualitative (QUAL) research. QUAN research is defined as an approach in which findings are derived from standard statistical procedures and other means of quantification in which data are numbers.
      • Boushey C.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      • Van Horn L.
      Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1.
      • Boushey C.J.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      Publishing nutrition research: A review of sampling, sample size, statistical analysis, and other key elements of manuscript preparation, Part 2.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      In contrast, QUAL research is defined as an approach that produces findings not derived from standard statistical procedures or other means of quantification, using a naturalistic approach that seeks to understand phenomena in uncontrolled, context-specific settings, in which data are not numbers, but text, audio, or visual. Key terms and definitions to this article are defined in Figure 1 and specific contrasting characteristics of QUAN and QUAL research are presented in Figure 2. Three previous articles in this statistical series have explicated these two paradigms in great detail and time will not be spent here reiterating the information.
      • Boushey C.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      • Van Horn L.
      Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1.
      • Boushey C.J.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      Publishing nutrition research: A review of sampling, sample size, statistical analysis, and other key elements of manuscript preparation, Part 2.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      However, a basic understanding of these two paradigms is needed to understand MMR.
      Figure 1Definitions of key mixed-methods research terms.
      TermDefinition
      Basic mixed-methods samplingA mixed-methods sampling that typically involves purposive sampling (ie, stratified purposive sampling, purposive random sampling) and/or a component of probability sampling (stratified or random).
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Between-strategyA data collection strategy in which qualitative and quantitative data are collected using the different strategies.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Cluster samplingA type of probability sampling in which the population is divided into groups on the basis of some shared characteristic (such as hospitals grouped by geographic region) and a random sample is drawn from each of these groups.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Concurrent timingWhen the researcher implements both the quantitative and qualitative strands during a single phase of a research study.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Convenience samplingA type of sampling in which the population selected is easily accessible to the researcher; available subjects are simply entered into the study without any attempt at randomization.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      ConvergenceWhen quantitative data are consistent with qualitative.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Convergent parallel designA mixed-methods design in which the researcher uses concurrent timing to implement the quantitative and qualitative strands during the same phase of the research process, prioritizes the methods equally, keeps the strands independent during analysis, and mixes the results during the researcher’s overall interpretation of the data.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Data saturationSaturation is the point where the range of qualitative ideas have been exhausted and no new additional information is being generated.
      • Krueger R.A.
      • Casey M.A.
      Focus Groups: A Practical Guide For Applied Research.
      DivergenceWhen quantitative and qualitative data conclusions are not consistent with one another.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Embedded designA mixed-method design in which the research collects and analyzes both quantitative and qualitative data within a traditional quantitative or qualitative design to enhance the overall design in some way.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Explanatory sequential designA two-phase mixed-methods design in which the researcher starts with the collection and analysis of quantitative data, followed by the collection and analysis of qualitative data to help explain the initial quantitative results.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Exploratory sequential designA two-phase mixed-methods design in which the researcher starts with the collection and analysis of qualitative data, followed by the collection and analysis of quantitative data to test or generalize the initial qualitative findings.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      InferencesConclusions or interpretations drawn from the separate quantitative and qualitative strands of a study as well as across the quantitative and qualitative stands, called meta-inferences.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Inference qualityThe internal validity of the QUAN
      QUAN=quantitative as main priority component or the first applied component.
      strand, whether the primary independent variable explains the dependent variable with little confounding, and the credibility of the QUAL
      QUAL=qualitative as main priority component or the first applied component.
      strand, whether the true perceptions of the participants has been captured.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Inference transferabilityThe degree to which the QUAN or QUAL research is generalizable to a larger population or other contexts and people.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Multilevel mixed-methods samplingA mixed-methods sampling procedure that involves the use of probability and purposeful sampling strategies at different levels of analysis.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Multiphase designA mixed-methods design that combines both the sequential and concurrent strands, collected over a period of time, and the implementation of distinct projects or phases within an overall program of study.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Nonprobability samplingSampling in which not every element of the population has an opportunity of being selected for the sample; the sample is not representative of the population and generalizations cannot be made to the population.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Parallel mixed-methods samplingA mixed-methods sampling procedure that involves selecting the unit of analysis simultaneously or with time lapse with both probability and purposeful sampling strategies (eg, one sampling procedure does not set the stage for the other).
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Probability samplingSampling in which each element of a population has an opportunity of being selected for the sample; its purpose is to obtain a sample that is representative of the population and from which generalizations to the population can be made.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Purposive samplingA type of nonprobability sampling in which the researcher consciously selects specific elements or subjects for inclusion in a study to ensure that the elements will have certain characteristics relevant to the study.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Quantitative researchApproach in which findings are derived from standard statistical procedures and other means of quantification. Experiments are conducted under controlled conditions in which data are numbers. The gold standard for this type of research is the randomized, controlled, clinical trial.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      QuantitizingA conversion analysis process that involves the transformation of qualitative data into codes and counts.
      Qualitative researchApproach that produces findings not derived from standard statistical procedures or other means of quantification. Defined as a naturalistic approach that seeks to understand phenomena in uncontrolled, context-specific settings, in which data are not numbers, but text, audio, or visual.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      QualitizingA conversion analysis processes that involves the transformation of quantitative data into qualitative data.
      Sequential mixed-methods samplingA mixed-methods sampling procedure that involves selecting the unit of analysis through the sequential use of probability and purposeful sampling strategies (eg, information from the first sample is required to draw the second sample).
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Snowball samplingA method whereby the names of prospective interview subjects for a statistical study are obtained from subjects already interviewed for the study.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      StrandA component of a mixed-methods study that encompasses the basic process of conducting quantitative or qualitative research: posing a question, collecting data, analyzing data, and interpreting results based on that data.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Transformative designA mixed-methods design that the researcher shapes within a transformative theoretical framework seeking to address the needs of a specific population and calling for change.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Evans B.C.
      • Coon D.W.
      • Ume E.
      Use of theoretical frameworks as a pragmatic guide for mixed methods studies: A methodological necessity?.
      TriangulationMethod of data validation that involves multiple methods, sources, and/or investigators to promote cross comparison of results.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      Within-strategyA data collection strategy in which qualitative and quantitative data are collected using the same strategy.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      a QUAN=quantitative as main priority component or the first applied component.
      b QUAL=qualitative as main priority component or the first applied component.
      Figure 2Characteristics of quantitative vs qualitative research.
      QuantativeQualitative
      Cause–effectProcesses
      Experimental, quasiexperimental, surveysPerceptions, perspectives, meaning
      Control confounding variables (validity)Contextual understanding
      OutcomesBarriers–benefits
      DeductiveNaturalistic, real-life settings
      Data: NumbersInductive
      Descriptive dataParticipant observation, interviews, focus groups, sight, sound
      Relationship between variablesNarratives, videos, transcripts, recordings
      Surveys, laboratory, anthropometric, written tools, clinical ratings, performanceThematic analysis
      This article begins with defining and describing the history and nature of MMR. Next, a conceptual overview of the six different MMR designs (Figure 3), data collection strategies, sample selection, data analysis, and inference processes are provided. Then, MMR concepts are integrated and nutrition-related scenarios are illustrated in real-world research contexts (Figure 4). The article concludes with summary recommendations for RDNs and aspiring nutrition researchers when conducting MMR.
      Figure thumbnail gr1
      Figure 3Diagrams of the six mixed-methods research designs. Timing or implementation of QUAN and QUAL data gathering can occur over a period of time (ie, sequentially) or at the same time (ie, concurrently). aQUAN=quantitative as main priority component or the first applied component. bQUAL=qualitative as main priority component or the first applied component. cqual=qualitative as the secondary component or the second component in the sequence. dquan=quantitative as the secondary component or the second component in the sequence.
      Figure 4Nutrition-related research scenarios for six different mixed-methods research designs.
      Mixed-methods research designResearch scenarioMixed-methods research questionsData collection methods and considerationsSampling approachData analysis and inference considerations
      Convergent parallelLow-income SNAP
      SNAP=Supplemental Nutrition Assistance Program.
      benefits and rural and urban mothers
      Are there quantitative and qualitative differences in the dietary behaviors, shopping patterns, and attitudes of SNAP benefits between rural vs urban mothers?QUAN
      QUAN=quantitative as main priority component or the first applied component.
      Food frequency questionnaire and close-ended attitude and shopping behavior questions.

      QUAL
      QUAL=qualitative as main priority component or the first applied component.
      Semistructured interview questions to explore dietary behaviors, shopping patterns, and attitudes, and determine differences in rural vs urban mothers.
      Parallel mixed-methods sampling:

      Sampling strands are defined on the outset of the study and include probability sampling of QUAN strand and purposeful sampling of QUAL strand based on locality.
      The QUAN and QUAL strand are first analyzed separately and then merged to show how the data are convergent or divergent. The intent is to compare different perspectives from QUAN and QUAL strands to derive a meta-inference of the study findings.
      Explanatory sequentialFruit and vegetable program in middle and high school studentsDo changes in foodservice effectively increase fruit and vegetable consumption in a middle and high school students across a school district?

      What are the perceptions and opinions among the middle and high school students on factor that influenced changes in fruit and vegetable consumption?
      QUAN Foodservice plate waste audits of middle and high school students’ fruit and vegetable consumption.

      qual
      qual=qualitative as the secondary component or the second component in the sequence.
      Focus groups of middle and high school students to explore opinions, barriers, and opportunities for choosing and consuming fresh fruits and vegetables at school.
      Sequential mixed methods sampling:

      Probability sampling of QUAN strand, followed by purposeful sampling of qual strand that unfolds during the course of the study and can only be determined after QUAN data are available. Participants in qual strand are drawn from the QUAN strand.
      The QUAN strand is analyzed and interpreted first. These data then inform the sampling and data collection procedures for the qual strand. The intent is to use the qual strand to provide a more in-depth understanding of the QUAN data.
      Exploratory sequentialMuslim registered dietitian nutritionistWhat are the characteristics and attitudes of Muslim dietitians living in the United States? What recruitment strategies would be most effective in enrolling Muslim dietitians in a survey research project?

      How do the characteristics and attitudes of Muslim dietitians living in the United States compare to other dietitians in the United States?
      QUAL Personal interviews of Muslim dietitians to uncover attitudinal issues and characteristics unique to these dietitians so a culturally appropriate questionnaire can be developed, as well as to answer questions about appropriate recruitment to engage Muslim dietitians in survey research.

      quan
      quan=quantitative as the secondary component or the second component in the sequence.
      A closed-ended survey designed to achieve a nationally representative sample of Muslim dietitians and compare characteristics and attitudes of Muslim dietitians living in the United States to other US dietitians.
      Basic mixed-methods sampling: Purposeful sampling of QUAL strand, followed by probability sampling of quan strand. Participants for both strands should be drawn from the same population. To avoid the introduction of bias and duplication of responses, participants in quan strand are not the same individuals included in the QUAL strand.The QUAL strand is analyzed and interpreted first. These data then inform the development of the quan measurement survey and the data sampling procedures.

      The intent is to use the QUAL strand to develop quan measurements with good psychometric properties and inform quan sampling procedures.
      Embedded-QUAN dominantDPP
      DPP=Diabetes Prevention Program.
      Will the DPP be effective when applied to inner city Hispanic men?

      Why or why not?
      QUAN Randomized clinical trial with pre- and posttest measurements of glycemic parameters among Hispanic men with impaired glucose tolerance.

      qual Videotaped personal interviews every month during the randomized trial regarding implementation of the DPP.
      Parallel mixed-methods sampling:

      Sampling procedures occur independently. Random sampling is used to identify QUAN strand sample through medical chart reviews. Purposeful sampling is used to identify participants for the qual strand.

      Participants in qual strand are drawn from the QUAN strand.
      For the QUAN strand, changes in glycemic measures are compared between the experimental and control group to assess the effectiveness of the program. The qual strand data are analyzed for themes related to program quality and implementation. The qual strand data are used to provide hypotheses explaining the effectiveness or ineffectiveness of the program.
      Embedded-QUAL dominantFood safety among employees of a large foodservice operationWhat are the perceptions, beliefs, and practices of employees regarding food safety?

      Are there difference by occupation level (ie, management, supervisory, or front line) within the company?
      QUAL Focus groups conducted based on occupational level within the company. Scripted discussion questions are used based on the themes of perceptions, beliefs, and practices related to food safety.

      quan Employees take an online survey regarding food safety knowledge and educational, cognitive, and reading levels.
      Basic mixed-methods sampling:

      Participants for QUAL strand are selected by stratified random sample, based on occupation level within the company. Participants in quan strand are the same participants from the QUAL strand.
      Narrative data from the QUAL strand analyzed for themes related to perceptions, beliefs, and practices. Data from the quan strand can be used to theorize the possible explanations for the emergent themes from QUAL strands.
      TransformativeFood insecurityCan a program for food-insecure people focused on increasing food-related self-efficacy improve their status?

      What factors influence why some individuals improve their food insecurity status and others do not?
      QUAN Food-insecure people participate in a 6-mo program to increase self-efficacy regarding the acquisition of food. Participants complete a pre- and postprogram closed-ended survey assessing changes (eg, food security status, self-efficacy, and knowledge).

      QUAL 4 Postprogram focus groups of participants, 2 with individuals who showed improvements in food insecurity and 2 with individual who did not show improvements. A food security discussion script is used and sessions are audiotaped.
      Sequential mixed-methods sampling:

      Convenience sample of QUAN strand, followed by purposeful sampling of qual strand that unfolds during the course of the study and can only be determined after QUAN data are available. Participants in QUAL strand are drawn from the QUAN strand.
      QUAN statistics are generated from the survey, including proportion of participants with improvement in food security and self-efficacy. These data then inform the sampling and data collection procedures for the QUAL strand. QUAL data are narratives generated from the audiotapes. Themes related to self-efficacy and resources are determined. Convergent parallel analysis is done to see whether QUAN and QUAL results are congruent.
      MultiphaseImplementation of cancer survivorship programWhat is the degree to which health-care organizations situated in rural areas have adopted and implemented evidence-based program for cancer survivors?

      What are the experience of organizations, providers, and cancer survivor patients that promote/discourage the uptake of evidence-based cancer survivorship programs?

      What characteristics of the health-care organization and providers promote organization-level sustainability and maintenance of evidence-based cancer survivorship programs?
      Within each of the 3 phases and 3 levels, a combination of QUAL and QUAN measurement approaches would be used, such as:

      organizational level, where organization-capacity quantitative survey (QUAN) and key informant interviews (QUAL) with system stakeholder are used;

      provider level, where process evaluation of implementation meeting minutes (QUAL) and ratings of program components (QUAN) are used; and patient level, where focus groups (QUAL) and self-reported (QUAN) improvements in quality of life and health behaviors are used.

      The emphasis (equal or unequal) of QUAL vs QUAN would vary within and across each phase.
      Multilevel mixed-methods sampling:

      Across each phase and level, a variety of sampling techniques would be used. For example, identification of organizations may include stratified random sampling, provider sampling may include purposeful sampling, and random sampling may be used to identify patients.
      This complex design requires an integrative mixed-methods research analytic approach. The goal is to promote the transfer and interchange of QUAL and QUAN forms of data within and across each phase of research, as well as within and among each sampling level (ie, organization, provider, and patient).
      a SNAP=Supplemental Nutrition Assistance Program.
      b QUAN=quantitative as main priority component or the first applied component.
      c QUAL=qualitative as main priority component or the first applied component.
      d qual=qualitative as the secondary component or the second component in the sequence.
      e quan=quantitative as the secondary component or the second component in the sequence.
      f DPP=Diabetes Prevention Program.

      The History and Nature of MMR

      Numerous definitions of MMR have been proffered, but the following will be useful for the discussion here: “It explores phenomena utilizing a combination of QUAL and QUAN methods in a single inquiry to gain a more complete perspective.”
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddlie C.
      • Tashakkori A.
      Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence.
      MMR is considered the third research paradigm in addition to QUAN and QUAL research. Although RDNs may be less familiar with MMR and it may seem to be a fairly new paradigm, mixed-methods approaches have been used in the distant past in fields as diverse as astronomy, sociology, and geology.
      • Maxwell J.A.
      Expanding the history and range of mixed methods research.
      Creswell,
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      a recent pioneer in MMR, has explicated the recent evolution of MMR since the 1950s. He delineates the periods as formative, paradigm debate, procedural development, advocacy and expansion, and reflective. During the formative period, MMR ideas were systematically tested in the field of sociology. During the paradigm debate period, it was questioned whether QUAL research was a true research paradigm. Once QUAL research was accepted as viable, scientists began to explore the specifics of the procedural steps in conducting MMR. During advocacy and expansion, application of MMR was explored in the context of addressing political and societal issues and solving problems in the world. Also, MMR use was embraced for use in most fields. The Journal of Mixed Methods Research was launched during this time. The current reflective period includes the consideration of nuances in study design and the consideration of theoretical and philosophical concerns. Investigators interested in exploring MMR in greater depth can access additional publications by prominent MMR researchers.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddlie C.
      • Tashakkori A.
      Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence.

      Creswell JW, Klassen AC, Plano Clark VL, Smith KC; National Institutes of Health. Best practices for mixed methods research in the health sciences. https://www.nursing.virginia.edu/media/Best_Practices_for_Mixed_Methods_Research.pdf. Published August 1, 2011. Accessed September 15, 2016.

      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.

      Why Combine Approaches?

      QUAL and QUAN are combined so weaknesses in one approach can be compensated by strengths in the other. QUAN approaches assume that truth and facts are objective, whereas QUAL assume that truth and facts have subjective elements based on the perceptions and meanings assigned by people. QUAN approaches can, with experiments and probability sampling, allow deduction of cause and effect and generalization of results to larger populations, whereas QUAL allows the investigator to consider context and processes in an inductive way. QUAL data promote a richer description of phenomena such as an individual’s perceptions, meanings, and overall experiences. Combining approaches leads to a more complete understanding of a phenomenon. For example, results from the Diabetes Prevention Program (DPP) demonstrated that through lifestyle change type 2 diabetes mellitus could be prevented.
      • Bishop J.
      • Irby M.B.
      • Isom S.
      • Blackwell C.S.
      • Vitolins M.Z.
      • Skelton J.A.
      Diabetes prevention, weight loss, and social support: Program participants' perceived influence on the health behaviors of their social support system.
      Suppose an investigator decides to replicate this program and test its effectiveness among inner-city Hispanic men. She collects baseline QUAN data, implements the study, and gathers QUAN outcome data. In contrast to previous studies, she finds the program to be ineffective at improving diabetes risk factors. This is clearly a QUAN approach. Had she added a QUAL component, conducting personal interviews with the participants at specific time points during the study, she could have gained significant insight about the men’s experiences throughout the program and their perceptions of the implementation of the program. In this way she could attempt to understand why the program did not work.
      MMR focuses on allowing the research problem and question to dictate the research design. QUAL and QUAN approaches are combined to best understand a phenomenon and ultimately solve problems.

      MMR Designs

      Why use an MMR approach? The most obvious answer is that the research problem or question requires it. More specific reasons are given below in the context of MMR research designs.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddlie C.
      • Tashakkori A.
      Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence.

      Creswell JW, Klassen AC, Plano Clark VL, Smith KC; National Institutes of Health. Best practices for mixed methods research in the health sciences. https://www.nursing.virginia.edu/media/Best_Practices_for_Mixed_Methods_Research.pdf. Published August 1, 2011. Accessed September 15, 2016.

      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
      • Bishop J.
      • Irby M.B.
      • Isom S.
      • Blackwell C.S.
      • Vitolins M.Z.
      • Skelton J.A.
      Diabetes prevention, weight loss, and social support: Program participants' perceived influence on the health behaviors of their social support system.
      • Bishop F.L.
      Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist perspective.
      • Evans B.C.
      • Coon D.W.
      • Ume E.
      Use of theoretical frameworks as a pragmatic guide for mixed methods studies: A methodological necessity?.
      • Migiro S.O.
      • Magangi B.A.
      Mixed methods: A review of literature and the future of the new research paradigm.
      • Morgan D.L.
      Practical strategies for combining qualitative and quantitative methods: Applications to health research.
      The research designs presented below are informed by the reasons MMR is used. The term strand as employed below is used in the current MMR literature to identify the QUAL or QUAN component of a design. In this section, research questions demanding MMR designs are posed and the Putting It All Together section further integrates real-world applications of MMR concepts.
      Six MMR designs have been identified to solve research problems.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.

      Creswell JW, Klassen AC, Plano Clark VL, Smith KC; National Institutes of Health. Best practices for mixed methods research in the health sciences. https://www.nursing.virginia.edu/media/Best_Practices_for_Mixed_Methods_Research.pdf. Published August 1, 2011. Accessed September 15, 2016.

      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
      • Bishop F.L.
      Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist perspective.
      • Ozawa S.
      • Pongpirul K.
      10 best resources on… mixed methods research in health systems.
      They are convergent parallel, explanatory sequential, exploratory sequential, embedded, transformative, and multiphase. These designs differ based on timing and priority of strands and match the primary reasons an MMR approach is used. Figure 3 depicts these MMR designs. On the design diagrams, the boxes indicate the application of the strands, whether QUAN or QUAL. The strand in caps (ie, QUAN or QUAL) represents either the main priority component or the first applied component. The strand not in caps (ie, quan or qual) represents the secondary component or the second component in the sequence. Also the timing or implementation of QUAN and QUAL data gathering can occur over a period of time (ie, sequentially) or at the same time (ie, concurrently). The research scenarios presented in this article are meant to illustrate application of the MMR approach, including appropriate sample selection and application of sound data analysis and interpretation techniques (Figure 4).
      In the convergent parallel design the QUAL and QUAN strands are of equal priority and applied independently at the same time. The goal is cross-validation (convergence, divergence, or triangulation) of data. Suppose an investigator wants to determine quantitative and qualitative differences in the dietary behaviors, shopping patterns, and attitudes toward Supplemental Nutrition Assistant Program (SNAP) benefits between rural vs urban mothers. The cross-sectional MMR design could include a QUAN questionnaire and a QUAL interview collected at approximately the same time points. Data from the questionnaire and interview would be mixed or blended to thoroughly understand the factors influencing dietary behaviors, shopping patterns, and attitudes, and determine whether the data from mixed sources confirm or disconfirm differences in rural vs urban mothers.
      For the explanatory sequential design, the QUAN strand is implemented first and is followed by a QUAL strand. Suppose a large, southern, urban school district launches a new fresh fruits and vegetables (F/V) program at the beginning of a school year and investigators want to explore two research questions: Do changes in foodservice effectively increase F/V consumption in middle and high school students across a school district? and, What are the perceptions and opinions among the middle and high school students on factors that influenced changes in F/V consumption? Throughout the school year, consumption data are collected via observational audits at middle and high schools across the district, including plate-waste analysis to determine whether F/V consumption goals are being met by students. The year-end report indicates that students are not meeting established goals for consuming more fresh F/V, yet there are also statistical differences across schools within the district as well as differences between middle and high school students. After reviewing the year-end report, the investigators conduct student focus groups to explain the differences in consumption patterns. The intent of this design is to use the QUAL methods to explain the QUAN results in more depth.
      With the exploratory sequential design, the QUAL strand is implemented first and then followed by a QUAN strand. This type of design is often used for survey development and to gain insights about effective recruitment of larger samples for survey purposes. For example, suppose investigators want to answer the following research questions: What are the characteristics and attitudes of Muslim RDNs living in the United States and what recruitment strategies would be most effective in reaching Muslim RDNs for a survey research project? and, How do the characteristics and attitudes of Muslim RDNs living in the United States compare with other RDNs in the United States? A QUAL strand can be implemented initially and include personal interviews collected via a snowball sample of Muslim subjects. The intent of this QUAL data is to uncover attitudinal issues and characteristics unique to Muslim RDNs to help develop a culturally appropriate questionnaire, as well as to understand appropriate recruitment methods to engage Muslim RDNs in survey research. The developed QUAN questionnaire can then be used in a full-scale survey research project. A central goal of an exploratory sequential design is to develop better QUAN instruments and measurements in unique samples and to improve the generalizability of QUAN findings.
      Embedded designs can be such that a QUAN strand is the priority with a smaller QUAL strand implemented at the same time but is less important than the QUAN strand. The DPP example given previously sans the preprogram QUAL component is an example in which this type of embedded design would be applied. Suppose a randomized trial is conducted with inner city Hispanic men with impaired glucose tolerance for 3 months in which the DPP is applied. The investigator desires to answer the question, Will the DPP be effective when applied to inner-city Hispanic men, and why or why not? Outcome data are collected, including glycosylated hemoglobin and fasting blood glucose levels. During the program, personal interviews are conducted every 2 weeks with members of the experimental groups asking about the implementation of the program. Suppose the outcome data reveal that the program was ineffective, the QUAL strand could inform investigators as to why.
      Another type of embedded design can involve the QUAL strand as the priority component and a less-prominent QUAN strand implemented at the same time. Four focus groups are conducted with a select group of foodservice employees of a large foodservice operation to assess perceptions, beliefs, and practices regarding food safety. The investigator would like to answer this question for this particular foodservice operation: What are the perceptions, beliefs, and practices of employees regarding food safety? At the same time, a survey of foodservice employees could be conducted to gather data about educational, cognitive, and reading levels to determine whether these factors might help explain the results of the QUAL strand.
      For a transformative MMR design, a social problem and the related accompanying theories become a context for the application of QUAL and QUAN strands. Social injustices, inequities, and barriers are explored with the ultimate goal to solve social problems. Any of the previous designs can be applied within the transformative design, so in a way it is less like a specific design and more of a philosophical or ideological approach. The ultimate goal of the transformative design is to test theories and solutions to social problems that are used to advocate for change. Suppose experts in US food insecurity believe that self-efficacy related to using public resources and choosing low-cost foods play a significant role in food insecurity in the US South. They believe that empowering people by increasing their knowledge and skills in accessing low-cost foods and food-related public resources could have a substantial influence on food insecurity status. They address the question, Can a program for food-insecure people focused on increasing food-related self-efficacy improve their status? QUAL and QUAN strands are combined to address these issues with the intention of ultimately developing solutions to effectively reduce the inequity of food insecurity.
      Lastly, a multiphase MMR design combines sequential and concurrent strands over a period of time to conduct a large, multidimensional project within an overall program of study. Suppose an investigator wants to answer the following research questions: What is the degree to which health-care organizations situated in rural areas have adopted and implemented an evidence-based program for cancer survivors? What are the experiences of organizations, providers, and cancer survivor patients that promote or discourage the uptake of evidence-based cancer survivorship programs? and, What characteristics of the organizations and providers promote organizational-level sustainability and maintenance of evidence-based cancer survivorship programs? These important, yet complex, questions would best be addressed in a multiphase MMR design. This design allows options for strand dominance, and flexibility in both QUAN and QUAL strands conducted concurrently within each phase and sequentially over a period of time.

      Defining the Research Question and Data Collection Strategies

      A thorough literature review and clearly articulated research questions will inform the MMR design as well as the type and order of data collected.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Regardless of the strand order, QUAL and QUAN measurement can both assume many forms. QUAN data can include objectively measured data, including anthropometric, biological, clinical, observational audit data, and knowledge, attitudes, behaviors, and cognitions survey data. QUAN research can include survey, epidemiologic, experimental, or quasiexperimental research designs. QUAL research approaches can consist of a variety of approaches, such as ethnographic research designs, personal interviews, participant observations, focus groups, document and social media analysis, and case studies. Both QUAN and QUAL survey research can include cross-sectional data or longitudinal data.
      Earlier articles published in Journal of the Academy of Nutrition and Dietetics illustrate the major data collection and analysis considerations when conducting QUAL or QUAN research that are independent of one another.
      • Boushey C.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      • Van Horn L.
      Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1.
      • Boushey C.J.
      • Harris J.
      • Bruemmer B.
      • Archer S.L.
      Publishing nutrition research: A review of sampling, sample size, statistical analysis, and other key elements of manuscript preparation, Part 2.
      • Harris J.E.
      • Gleason P.M.
      • Sheean P.M.
      • Boushey C.
      • Beto J.A.
      • Bruemmer B.
      An introduction to qualitative research for food and nutrition professionals.
      Therefore, the purpose of this article is to focus on key considerations for MMR data collection strategies. These data collection strategies can be generally categorized as within-strategy or between-strategy collections.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Within-strategy refers to gathering both QUAL and QUAN data using the same data collection strategy. An interview that is designed to assess both QUAN and QUAL data on low-income mothers receiving SNAP benefits is one example. Between-strategy involves gathering QUAL and QUAN data using more than one data collection strategy. A research design including both school plate waste audits of F/V consumption and focus groups to explore students’ perceptions of food is an example of between-strategy data collection. Also, MMR data collection can occur at a single level or at multiple levels.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Data collection levels are often referred to as individual, interpersonal, organizational, community, and policy levels. Both of the prior examples wherein data collection was restricted to the low-income mother or the middle-school student are examples of an individual- and single-level strategy. However, the school foodservice example could be expanded to include multiple levels within the school, such as adding interviews of foodservice organizational staff. This would be a multilevel data collection strategy. Also, as described further below, the cancer survivorship program is an example of a multilevel data collection strategy because it integrates data collected at the individual patient level, the provider level, and the health care organization level (see Figure 4).
      The potential combination of research questions, appropriate MMR designs, and measurement combinations are countless. To promote the best interpretation of an MMR question, the challenge is to developed well-conceptualized and integrated research questions, followed by careful consideration for sampling in each strand, rigorous data collection methodology, and transparent emphasis for each strand (equal or unequal), and to apply analysis techniques that ensure the most appropriate mixing or blending of data.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Johnson R.B.
      • Onwuegbuzie A.J.
      • Turner L.A.
      Towards a definition of mixed methods research.

      Sampling Strategies

      Several sampling typologies are used in nutrition and dietetics research, including probability, purposive, snowball, and convenience sampling.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      In brief, probability sampling is primarily used in QUAN research and includes the random selection of subjects from a population, with the goal of achieving representativeness of the population (eg, simple random sampling and stratified random sampling). Selection of the cases involves selecting a large enough sample size to establish representativeness and ensure adequate statistical power to draw conclusions. Purposive sampling is often used in QUAL research and involves selecting subjects based on a specific purpose, rather than randomly. The subjects are often selected on expert judgment and typically include fewer subjects as compared with probability sampling. There are no steadfast rules for sample size in QUAL research; rather, the goal is to achieve data saturation.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Krueger R.A.
      • Casey M.A.
      Focus Groups: A Practical Guide For Applied Research.
      Saturation is the point where the range of QUAL ideas have been exhausted and no new additional information is being generated.
      • Krueger R.A.
      • Casey M.A.
      Focus Groups: A Practical Guide For Applied Research.
      Convenience sampling draws subjects that are most easily accessible and willing to participate, yet may not be representative of the population or the most appropriate to answer the research questions. This type of sampling often results in biased results. The classic trade-off is that probability sampling leads to a breadth of information from a large, representative sample that promotes generalizability of study findings. On the contrary, purposive sampling produces greater depth of information from a smaller selection of subjects, lacking generalizability. In general, the sample size from the QUAL data collection will typically be smaller than from the QUAN data collection. However, it is important to acknowledge that quantitative research is sometimes based on purposeful and/or small samples, and QUAL research occasionally based on large, purposefully selected samples.
      Mixed-methods sampling uses a combination of probability and nonprobability (eg, purposive sampling).
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      All of the assumptions, strengths, and limitations of basic sampling still exist, yet the complexity increases due to the mixed-methods approach. The primary advantage of using MMR sampling is that deficits and weaknesses of one sampling procedure are compensated by the strengths of the other. A notable disadvantage with MMR sampling is the added time and resources to sample appropriately. There are four general types of MMR sampling: basic, parallel, sequential, and multilevel.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      These sampling approaches are defined and illustrated in Figure 1 and described in more detail below; however, a basic overview of data analysis and inference of MMR is first necessary.

      Analysis Procedures for MMR Designs

      MMR analysis is an iterative process with the goal of cross-validating, or combining QUAL and QUAN data.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Collins K.M.T.
      • Onwuegbuzie A.J.
      • Jiao Q.G.
      A mixed methods investigation of mixed methods sampling designs in social and health science research.
      • Teddlie C.
      • Yu F.
      Mixed-methods sampling a typology with examples.
      Tied directly to the type of MMR design, data analysis procedures can be categorized as parallel, conversion, sequential, and integrated.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Each procedure includes the combining of QUAL and QUAN data.
      Parallel analysis is the most widely used.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Greene J.C.
      In this approach, the QUAL and QUAN analysis are two separate processes. Analysis of each strand is conducted independently and conclusions are drawn, after which the data from the strands are considered together. Often the goal is triangulation, the cross-validation of results leading to meta-inferences about convergence or divergence of QUAL and QUAN conclusions. Two examples include side-by-side comparisons and joint display. In the side-by-side comparison approach, the QUAN statistical data are presented first, followed by QUAL findings that confirm or disconfirm the statistical data. A final summary statement is made talking about convergence or divergence of data. For instance, the QUAN strand may indicate that an intervention was ineffective at improving cognitions (eg, attitudes and perceived behavioral control), but the QUAL strand may indicate the opposite. This is an example of divergence. There is a second type of side-by-side comparison table that can be created. Three columns are created: themes, QUAL data related to each theme, and QUAN data related to each theme. With joint display of data, information is typically in the form of a table. For instance, themes from the QUAL data are displayed in a table (eg, differences in rural vs urban dietary, shopping, and attitudes), along with QUAN data (eg, means, standard deviations, and ranges of survey response) in a contingency table format. The ultimate goal is to form meta-inferences that explain the convergence or divergence of QUAL and QUAN findings.
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Greene J.C.
      Conversion analysis involves processes when QUAL data are transformed into codes and counts (quantitizing) or QUAN data are converted to QUAL data (qualitizing).
      • Creswell J.
      • Plano Clark V.
      Designing and Conducting Mixed Methods Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      One example of quantitizing includes converting QUAL themes into numeric codes (eg, 0 and 1 or 1, 2, and 3). These coded data can then be linked or merged with the original QUAN strand and analyzed using simple descriptive statistics or more complex interferential statistical tests. One example of qualitizing data includes the process of producing QUAL themes based on numerical survey data. In the convergent parallel design example, the questionnaire data (means and percentages) from the QUAN strand about dietary behaviors, shopping patterns, and attitudes toward SNAP benefits can be converted to qualitative themes and compared with themes from the QUAL strand.
      In the sequential analysis, the strands are dependent on one another, such that the analysis and results of one strand are used to plan the next strand, regardless of whether the QUAL or QUAN strand is initiated first.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Greene J.C.
      Therefore, the phases of the analysis evolve as the study unfolds. Generally the databases are not merged in a sequential design. In an explanatory sequential design, the QUAN data are used to inform the collection of QUAL data. In an exploratory sequential design, the QUAL data are used to inform the collection of QUAN data. In both cases the second phase QUAN and QUAL data may be used to explain the data in the first phase. Therefore, in a sequential analysis, the findings of each strand are first presented in the results, then interpretation of how one strand informs the other typically occurs in the discussion section.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Greene J.C.
      For the more advanced MMR designs, such as embedded, transformative, and multiphase, an integrative mixed-data analysis process is often needed.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Characterized as interdependent, iterative, and reciprocal, the integrative analysis process is the interactive mixing of QUAL and QUAN data. The integrative analysis can include any combination of the prior-mentioned analysis processes (ie, parallel, conversion, or sequential) and the major underlying concepts guiding the analysis remain the same. When integrative mixed data analysis is done successfully, there will be more comprehensive answers to research questions.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Consistent with all MMR analysis approaches, interpretations of the convergence and divergence of MMR data are critical. Convergence (ie, consistency among quantitative and qualitative data) is not the ultimate goal of MMR research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      • Greene J.C.
      Generating divergent conclusions (ie, inconsistency among quantitative and qualitative data) between data sources can be of great value and one of the major advantages of MMR. MMR designs rarely produce precise convergence and divergence, and it is the investigators’ responsibility to further analyze their data to offer explanations.
      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.

      The Inference Process for MMR

      Arguably the most important aspect in an MMR study is the inference process, or the practice of deriving meaning from the results.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Creswell J.W.
      Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      The overarching foundational inference concepts include inference quality and inference transferability.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
      Inference quality refers to the internal validity—that is, whether the primary independent variable explains the dependent variable with little confounding—of the QUAN strand, and the credibility—that is, whether the true perceptions of the participants have been captured—of the QUAL strand. Inference transferability refers to external validity, or the degree to which the QUAN or QUAL research is generalizable to a larger population or other contexts and people. Although a complete description of the complexities of MMR inference is beyond the scope of this article, other sources elaborate on the multidimensional aspects of MMR inference in more detail.
      • Tashakkori A.
      • Teddlie C.
      Sage Handbook of Mixed Methods in Social and Behavioral Research.
      • Teddie C.
      • Tashakkori A.
      Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.

      Putting It All Together

      In this section, previously mentioned MMR concepts will be integrated (a summary is provided in Figure 4). For the convergent parallel design, both the QUAN and QUAL data are collected during the same time period. Assessing rural and urban differences for dietary behaviors, shopping patterns, and attitudes among low-income families receiving SNAP benefits, using both QUAN food frequency questionnaires and surveys as well as interview methodology, is one example. A mixed-methods parallel sampling approach is a good fit for this research. To address QUAN differences, a probability sample with an adequate number of participants and representativeness to inform the generalizability of findings is needed. For example, suppose 150 rural mothers and 150 urban mothers are needed to find meaningful statistical differences between groups and ensure a large enough sample to generalize findings. Because the intent of QUAL interviews is to achieve in-depth perspectives, it would not be practical to interview all 300 mothers. In a convergent parallel design, the QUAL sample is usually taken from the larger QUAN sample. For example, of the 150 urban and rural mothers surveyed, about 25 each may be purposefully sampled (eg, different socioeconomic status, diverse racial or ethnic groups) to participate in the QUAL interview. The QUAL sampling is limited to a sample that can adequately achieve data saturation. Each strand of data is analyzed separately and then merged or converged to derive inferences within each strand of the study. It is the investigators’ responsibility to resolve and offer explanations related to convergent and divergent findings. For example, the QUAN analysis may have revealed few statistical differences in dietary behaviors or attitudes among rural and urban mothers receiving SNAP benefits, but the QUAL analysis may have revealed clear thematic differences in the perceptions and behavioral challenges based on locality.
      For the explanatory sequential design, the primary intent is to use QUAL data to help explain and build upon the QUAN findings. Therefore, data collection occurs in two distinct phases. The first phase involves rigorous QUAN methods, analysis, and interpretation. Subsequently a second phase of QUAL research methods is used. The use of observational plate waste audit data and analysis, followed by student focus groups to explain the low consumption of fresh F/V, is an example of an explanatory sequential design and a sequential mixed-methods sampling approach. Suppose a large, southern, urban school district had 30 middle and high schools. Based on available resources and statistical requirements, they randomly selected 10 middle and 10 high schools for the observational audits and included 50 student plate-waste audit points at each of the three times throughout the school year. The sampling plan and script for the QUAL focus groups can only be finalized after the plate-waste data are analyzed and available. Given that there were statistical differences across schools between middle and high school students, the researchers would want to purposively sample schools to further explore these differences. After reviewing the year-end report, the investigators conduct eight student focus groups (eight to 10 students each) to explain the differences in consumption patterns. Based on QUAN differences, researchers may have chosen to conduct half of the focus groups with middle school students and half with high school students, as well as half within schools that showed improvements in consumption patterns and half with no improvements. Because the goal of this explanatory sequential design is to explain QUAN variables and mechanisms, the same individuals should be included in both strands. The intent of this design is to use the QUAL methods to explain the QUAN results in more depth. One challenge with this type of MMR design can be prioritizing what aspects of QUAN data to follow-up on qualitatively. In this type of design, QUAN and QUAL data are always analyzed separately, and the QUAN findings can be used to inform both the types of QUAL questions to ask, as well as the sampling procedure. In the data interpretation, the QUAN audit data should be presented first (eg, results from plate waste study and differences by schools and student type) followed by QUAL themes that help explain the differences. For example, among the schools with higher consumption, focus group findings may have revealed the importance of increased variety and quality of fresh F/V offerings, offering dips with the F/V, and serving the fruit cut and/or peeled or sectioned. Overall conclusions in explanatory sequential MMR designs should specify how the QUAL results help explain the QUAN findings.
      An important goal of the exploratory sequential MMR design is to use QUAL findings to improve the psychometric instrument properties and sampling procedures of QUAN methods. There are two distinct phases of data collection, starting with QUAL data collection, analysis, and interpretation and followed by a second phase of QUAN research methods. The Muslim RDN scenario demands an exploratory sequential MMR design because an adequate instrument to measure intended concepts is not available. This scenario would include a basic mixed-methods sampling plan; for example, a convenience sample (eg, snowball sample) may be used for the QUAL phase and a probability sample for the QUAN phase. Although participants from both strands should be drawn from the same population, participants in the QUAN strand are not the same individuals included in the QUAL strand. This is necessary to avoid the introduction of bias and duplication of responses. For example, a QUAL strand can be initially implemented and include personal interviews collected from 25 Muslim subjects. Once the QUAL data are analyzed, they are used to inform the subsequent QUAN phase, including adequate development of an instrument to measure the characteristics and attitudes of Muslim RDNs living in the United States. Subsequently, the QUAN phase would be executed to achieve a nationally representative sample of Muslim and US RDNs. The sample should be large enough to draw statistical inference between groups and promote the generalizability of findings. In this type of design, strands are always analyzed separately, and the QUAL findings are used to inform both the types of QUAL instrument development and sampling approach. Comparing the strand sources is not appropriate because they are typically from different samples; however, it is the investigators’ responsibility to discuss the prioritization of QUAL themes and decisions applied to inform the development of QUAN concepts. This design is often referred to as a three-phrase process, including exploratory QUAL inquiry, QUAN instrument development, and QUAN instrument administration to a population.
      For embedded designs, either the QUAN or QUAL strand is dominant. In the case that the QUAN strand is dominant and of primary importance, a recessive concurrent QUAL strand is included to enrich and supplement the QUAN strand. When the QUAL strand is dominant, the QUAN strand assumes the recessive role. Using the previously explicated diabetes example conducted with inner-city Hispanic men with impaired glucose tolerance, the QUAN strand is dominant and the QUAL strand recessive (see Figure 4). Both probability and purposeful sampling are used simultaneously in this parallel mixed-methods sampling. In this case, 100-inner city Hispanic men with impaired glucose tolerance were identified through medical record reviews and randomized to two groups of 50. One group received the DPP and the other newsletters, both for 12 months. Both groups had pre- and post-program laboratory measurements of fasting blood glucose and glycated hemoglobin. By comparing post- and pretest values, it can be determined whether the program is effective compared with the newsletter control group. The randomized trial with the comparison of the quantitative laboratory measurements is the QUAN strand. Concurrent with the program and defined on the outset of the study, 10 individuals for the experimental group were chosen by purposeful sampling (ie, five who are aged 20 to 39 years and five who are aged 40 to 59 years) to participate in personal interviews every month that are videotaped (for a total of 12 sessions). Program participants were questioned about implementation of the program and asked to give narratives about the quality of instruction, the time of day of the program, accessibility to the program, quality of program materials, aesthetics about the location, and perceptions of other participants. Videotapes can then be evaluated and written narratives produced that can be analyzed for themes. In addition, because interviews are conducted every month, the evolutionary progression of the program can be analyzed. The primary concern of investigators is whether the program is effective. The secondary concern is about the quality of the program and whether it can produce hypotheses about why the program was or was not effective. The QUAL data are used to supplement the QUAN data.
      The second type of embedded design has a dominant QUAL strand and a recessive QUAN strand. The example presented earlier involved addressing the food safety perceptions, beliefs, and practices of employees of a large, foodservice operation. Suppose this operation has 10 locations with 20 upper management, 30 supervisory, and 200 front-line workers. Four separate 2-hour focus groups are conducted: two with upper management, two with supervisors, and four with front-line workers. Employees are chosen by stratified random sampling, by occupation level, to populate each focus group. These focus groups are videotaped and led by trained facilitators. A discussion script is used with questions about food safety perception, beliefs, and practice. At the end of the focus group phase, videotapes are translated into written narratives. These narratives are analyzed for themes with qualitative analysis software. The recessive QUAN strand consists of a survey testing the knowledge and determining the educational, cognitive, and reading levels of the employees. Before the focus groups, employees selected for focus groups are also asked to fill out an online quantitative survey deployed through the company website. QUAN knowledge scores and educational, cognitive, and reading levels are determined for the sample. These data are used to attempt to provide hypothesized explanations for the QUAL themes. This example illustrates a basic mixed-methods sampling protocol.
      The transformative mixed methods design is used to address real social problems in an attempt to understand social inequities, power relationships, and barriers to social justice. As presented earlier, any of the other MMR designs can be applied in a transformative way, as long as the goal of the research is to address social problems, seek potential solutions, and advocate for change. An example was proposed earlier related to food insecurity. Suppose experts in US food insecurity believe that self-efficacy related to using public resources and choosing low-cost foods play a significant role in food insecurity in the southern United States. They believe that empowering people by increasing their knowledge and skills in accessing low-cost foods and food-related public resources could have a substantial influence on food insecurity status. Investigators conveniently choose one low-socioeconomic county in their state. They worked with the local social services agency to identify food-insecure people and enroll 50 participants into the program and collect baseline quantitative surveys (ie, food security status, self-efficacy, use of public resources, and knowledge of low-cost food items). They conduct the 6-month program free of charge at the local community center. They provided both transportation and child care. At the conclusion of the program, the quantitative survey is re-administered. This is the QUAN strand. QUAN descriptive statistics are produced that document the outcomes of the program. Two weeks later, four focus groups are conducted: two with participants who showed improvements in food insecurity, and two with participants who did not show improvements. With this QUAL strand, participants are asked open-ended questions about food insecurity-related issues. The focus groups are audiotaped and narratives are generated for theme analysis. The investigators compare the survey and focus group results to determine to what degree there is convergence or divergence of results. This is a convergent parallel design conducted within a transformative approach, where the investigators ultimately intend to develop potential interventions to improve food insecurity that can have more widespread deployment.
      The multiphase MMR design related to the scenario regarding cancer survivorship programs would demand multilevel mixed-methods sampling. Phase 1 may include a stratified random sample of health centers across eight rural regions of the United States. Based on Phase 1 results, Phase 2 could include an implementation intervention to promote participation in cancer survivorship programs and include a purposeful sample of organizations and providers, but a random sample of survivors. Phase 3 would include a sequential mixed-methods sample of those involved in Phase 2, where emergent Phase 2 data were used to derive the sampling plan. Within each of the three phases and accompanying research questions, a combination of QUAL and QUAN measurement approaches would vary and each would be specific to organization (eg, organization capacity quantitative survey and key informant interviews with system stakeholder), provider (eg, process evaluation of implementation meeting minutes and ratings of program components), and patient (eg, focus groups and self-reported improvements in quality of life and health behaviors) levels. Given that the research questions occur in a hierarchy setting, multilevel analyses are necessary. For example, the units of analysis are nested within another (eg, cancer survivor patients are linked to a provider, whereas patients and providers are within a health-care organization). To promote the transfer and interchange of QUAL and QUAN forms of data within and among each phase, this complex design would demand an integrative MMR analytic approach.

      Summary Recommendations

      This article provides an overview of MMR research that can be applied in the research of RDNs and aspiring nutrition researchers. Overall summary recommendations when conducting MMR research include:
      • Be clear in the selection and description of your MMR design. Different integrated research questions demand different MMR designs. Understanding the intent, strengths, and limitations of each MMR design at the start of the research project will promote inference quality and inference transferability at the conclusion.
      • Be transparent in the emphasis for all QUAN and QUAL strands. Equal emphasis on each strand in not a requirement in MMR. However, it is necessary to apply robust methods and analytical strategies for all strands. Through careful planning, researchers can avoid having the MMR design appear as an afterthought or one strand being substantially weaker in design and execution. For several MMR designs, planning must be a flexible and ongoing process because one phase of the study is necessary to inform the next phase.
      • Recognize the strengths and limitations in your sampling approach. A primary consideration in the sample size determination for the QUAN strand is sufficient analytic power to detect statistical relationships and/or changes over time. Alternatively, the primary considerations for QUAL data analysis are reliant on issues related to data saturation. Be realistic and pragmatic about the scope of the research and the availability of resources to execute the best possible sampling plan.
      • Apply appropriate methodologies when mixing or blending your data. There are numerous options for mixing data, including, but not limited to, side-by-side comparisons, joint display, conversion (quantitizing and qualitizing), and integrative approaches. One of the best ways to become familiar with strengths and limitations for mixing data is to critically review other empirical MMR manuscripts, such as those found in Journal of Mixed Methods Research. Examples of multistage,
        • Christ T.W.
        A recursive approach to mixed methods research in a longitudinal study of postsecondary education disability support services.
        exploratory sequential,
        • Stoller E.P.
        • Webster N.J.
        • Blixen C.E.
        • et al.
        Alcohol consumption decisions among nonabusing drinkers diagnosed with Hepatitis C: An exploratory sequential mixed methods study.
        • Shahriari M.
        • Mohammadi E.
        • Fooladi M.M.
        • Abbaszadeh A.
        • Bahrami M.
        Proposing codes of ethics for Iranian nurses: A mixed methods study.
        embedded,
        • Clark V.L.
        • Schumacher K.
        • West C.
        • et al.
        Practices for embedding an interpretive qualitative approach within a randomized clinical trial.
        • Leal I.
        • Engebretson J.
        • Cohen L.
        • et al.
        An exploration of the effects of Tibetan yoga on patients’ psychological well-being and experience of lymphoma: An experimental embedded mixed methods study.
        concurrent parallel,
        • Jang E.E.
        • McDougall D.E.
        • Pollon D.
        • Herbert M.
        • Russell P.
        Integrative mixed methods data analytic strategies in research on school success in challenging circumstances.
        explanatory sequential,
        • Hesse-Biber S.
        Gender differences in psychosocial and medical outcomes stemming from testing positive for the BRCA1/2 genetic mutation for breast cancer: An explanatory sequential mixed methods study.
        and transformative
        • Barnhardt C.L.
        • Reyes K.
        • Rodriguez A.V.
        • Ramos M.
        A transformative mixed methods assessment of educational access and opportunity for undocumented college students in the southeastern United States.
        designs are available for review.
      • Critically interpret the convergence and divergence of your MMR data. MMR designs rarely produce precise convergence and divergence, and it is the investigators’ responsibility to further analyze their data to resolve and offer explanations. Anticipate that MMR data interpretation is an iterative process and typically requires more time and resources than a study focused exclusively on either QUAN or QUAL methods.
      • As a nutrition and dietetics practitioner, consider using MMR designs. Literature searches yield a dearth of nutrition or dietetics-related articles using a variety of MMR designs. There is a definite need for more MMR-related literature.

      Acknowledgements

      The authors thank the Journal of the Academy of Nutrition and Dietetics Statistical Committee, and especially thank committee members Philip Gleason, PhD, Mary Naglak, PhD, RD, LDN, and Judith Beto, PhD, RD, LDN, FAND, for reviewing and providing comments that greatly improved the manuscript.

      References

        • Boushey C.
        • Harris J.
        • Bruemmer B.
        • Archer S.L.
        • Van Horn L.
        Publishing nutrition research: A review of study design, statistical analyses, and other key elements of manuscript preparation, Part 1.
        J Am Diet Assoc. 2006; 106: 89-96
        • Boushey C.J.
        • Harris J.
        • Bruemmer B.
        • Archer S.L.
        Publishing nutrition research: A review of sampling, sample size, statistical analysis, and other key elements of manuscript preparation, Part 2.
        J Am Diet Assoc. 2008; 108: 679-688
        • Harris J.E.
        • Gleason P.M.
        • Sheean P.M.
        • Boushey C.
        • Beto J.A.
        • Bruemmer B.
        An introduction to qualitative research for food and nutrition professionals.
        J Am Diet Assoc. 2009; 109: 80-90
        • Creswell J.
        • Plano Clark V.
        Designing and Conducting Mixed Methods Research.
        2nd ed. Sage Publications, Thousand Oaks, CA2011
        • Tashakkori A.
        • Teddlie C.
        Sage Handbook of Mixed Methods in Social and Behavioral Research.
        2nd ed. Sage Publications, Thousand Oaks, CA2010
        • Teddlie C.
        • Tashakkori A.
        Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence.
        Am Behav Sci. 2012; 56: 774-788
        • Maxwell J.A.
        Expanding the history and range of mixed methods research.
        J Mix Methods Res. 2015; 10: 12-27
      1. Creswell JW, Klassen AC, Plano Clark VL, Smith KC; National Institutes of Health. Best practices for mixed methods research in the health sciences. https://www.nursing.virginia.edu/media/Best_Practices_for_Mixed_Methods_Research.pdf. Published August 1, 2011. Accessed September 15, 2016.

        • Creswell J.W.
        Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.
        Sage Publications, Thousand Oaks, CA2013
        • Bishop J.
        • Irby M.B.
        • Isom S.
        • Blackwell C.S.
        • Vitolins M.Z.
        • Skelton J.A.
        Diabetes prevention, weight loss, and social support: Program participants' perceived influence on the health behaviors of their social support system.
        Fam Community Health. 2013; 36: 158-171
        • Bishop F.L.
        Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist perspective.
        Br J Health Psychol. 2015; 20: 5-20
        • Evans B.C.
        • Coon D.W.
        • Ume E.
        Use of theoretical frameworks as a pragmatic guide for mixed methods studies: A methodological necessity?.
        J Mixed Methods Res. 2011; 5: 276-292
        • Migiro S.O.
        • Magangi B.A.
        Mixed methods: A review of literature and the future of the new research paradigm.
        Afr J Bus Manage. 2011; 5: 3757-3764
        • Morgan D.L.
        Practical strategies for combining qualitative and quantitative methods: Applications to health research.
        Qual Health Res. 1998; 8: 362-376
        • Ozawa S.
        • Pongpirul K.
        10 best resources on… mixed methods research in health systems.
        Health Policy Plan. 2014; 29: 323-327
        • Teddie C.
        • Tashakkori A.
        Foundations of Mixed Methods Research: Integrating Quantiative and Qualitative Apporaches in the Social and Behavioral Sciences.
        Sage Publications, Inc, Thousand Oaks, CA2009
        • Johnson R.B.
        • Onwuegbuzie A.J.
        • Turner L.A.
        Towards a definition of mixed methods research.
        J Mix Methods Res. 2007; 1: 112-133
        • Krueger R.A.
        • Casey M.A.
        Focus Groups: A Practical Guide For Applied Research.
        4th ed. Sage, Thousand Oaks, CA2009
        • Collins K.M.T.
        • Onwuegbuzie A.J.
        • Jiao Q.G.
        A mixed methods investigation of mixed methods sampling designs in social and health science research.
        J Mix Methods Res. 2007; 1: 267-294
        • Teddlie C.
        • Yu F.
        Mixed-methods sampling a typology with examples.
        J Mix Methods Res. 2007; 1: 77-100
        • Greene J.C.
        Mixed Methods in Social Inquiry. Vol. 9. John Wiley & Sons, New York, NY2007
        • Christ T.W.
        A recursive approach to mixed methods research in a longitudinal study of postsecondary education disability support services.
        J Mix Methods Res. 2007; 1: 226-241
        • Stoller E.P.
        • Webster N.J.
        • Blixen C.E.
        • et al.
        Alcohol consumption decisions among nonabusing drinkers diagnosed with Hepatitis C: An exploratory sequential mixed methods study.
        J Mix Methods Res. 2009; 3: 65-86
        • Shahriari M.
        • Mohammadi E.
        • Fooladi M.M.
        • Abbaszadeh A.
        • Bahrami M.
        Proposing codes of ethics for Iranian nurses: A mixed methods study.
        J Mix Methods Res. 2016; 10: 352-366
        • Clark V.L.
        • Schumacher K.
        • West C.
        • et al.
        Practices for embedding an interpretive qualitative approach within a randomized clinical trial.
        J Mix Methods Res. 2013; 7: 219-242
        • Leal I.
        • Engebretson J.
        • Cohen L.
        • et al.
        An exploration of the effects of Tibetan yoga on patients’ psychological well-being and experience of lymphoma: An experimental embedded mixed methods study.
        J Mix Methods Res. 2016; ([published online ahead of print May 2, 2016])https://doi.org/10.1177/1558689816645005
        • Jang E.E.
        • McDougall D.E.
        • Pollon D.
        • Herbert M.
        • Russell P.
        Integrative mixed methods data analytic strategies in research on school success in challenging circumstances.
        J Mix Methods Res. 2008; 2: 221-247
        • Hesse-Biber S.
        Gender differences in psychosocial and medical outcomes stemming from testing positive for the BRCA1/2 genetic mutation for breast cancer: An explanatory sequential mixed methods study.
        J Mix Methods Res. 2016; ([published online ahead of print June 29, 2016])https://doi.org/10.1177/1558689816655257
        • Barnhardt C.L.
        • Reyes K.
        • Rodriguez A.V.
        • Ramos M.
        A transformative mixed methods assessment of educational access and opportunity for undocumented college students in the southeastern United States.
        J Mix Methods Res. 2016; ([published online ahead of print June 14, 2016])https://doi.org/10.1177/1558689816652764

      Biography

      J. Zoellner is an associate professor, Department of Public Health Sciences, School of Health Sciences, University of Virginia, Charlottesville.
      J. E. Harris is a professor and the chair, Nutrition Department, West Chester University of Pennsylvania, West Chester.