Advertisement

Measures Used with Populations with Food Insecurity: A Call for Increased Psychometric Validation

      Keywords

      How do researchers know what they are measuring when they use questionnaires? Confidence in the ability of measures to appropriately capture intended constructs for a particular population is essential to scientific rigor and a challenge for the social sciences. Parnarouskis and colleagues
      • Parnarouskis L.
      • Gearhardt A.N.
      • Mason S.E.
      • et al.
      Association of food insecurity and food addiction symptoms: a secondary analysis of two samples of low-income female adults.
      bring up a critical point in their Discussion about the limitations of using a measure for which the validation samples may not have adequately represented specific groups of interest. We appreciate the opportunity to continue this important conversation because it is a topic that is of high importance to the study of food insecurity and eating-related behaviors.
      It is common for researchers to use psychometrically validated measures in populations that were not systematically included (or were even systematically excluded) in measures’ initial validation studies. This occurs because measures may be validated in samples with lower diversity in important variables like race, ethnicity, socioeconomic status, gender, and sexual orientation. Further, even when there is diversity represented within an overall sample, studies may be underpowered to test measure performance differences in sample subgroups. Consequently, measures are often used to assess questions for populations in which they have yet to be validated. It is often not practical for measure developers to conduct an in-depth psychometric validation using all potential populations before releasing a measure. However, it is critical that measurement validation work continues beyond the initial validation to establish that measures function appropriately in populations of interest, which will increase confidence in conclusions drawn from research.
      For a measure to have adequate psychometric properties, it must be both reliable and valid. Reliability refers to the extent to which measurement scores replicate across different populations, across time, and across different components of the measure itself. Validity is the extent to which a psychometric tool measures what it was designed to measure. When researchers develop a new measure, they often conduct an initial study in participants from the general population to establish internal consistency, test–retest reliability, and construct validity, as well as to identify the factor structure.
      • Furr M.
      Scale Construction and Psychometrics for Social and Personality Psychology.
      The measure may then be tested within a specific subpopulation of interest. Below, we describe some challenges with applying measures to novel populations for whom such measures have not been psychometrically validated and—to help equip researchers with the necessary psychometric toolbox—outline some methodological approaches to determine whether measures perform similarly across novel populations.

      Psychometric Validation Is Needed for Measures Assessing Populations with Food Insecurity

      One must be cautious in interpreting study results when using measures in populations for whom the psychometric validity of such measures has not been established. This seems particularly important when measuring eating-related constructs in people with food insecurity because their eating patterns may deviate from those of the general population for reasons other than those intended to be captured by a given measure. Furthermore, caution is warranted to avoid overpathologizing natural physiological, behavioral, and emotional responses to the stressor of food insecurity.
      There are conceptual reasons why it would be critical to validate eating-related measures (eg, food addiction, and eating disorder symptoms) in populations with food insecurity. Because the experience of food insecurity involves—by definition—disruption of food intake or eating patterns because of lack of money and other resources,
      • Anderson S.A.
      Core indicators of nutritional state for difficult-to-sample populations.
      ,
      • Pinstrup-Andersen P.
      Food security: definition and measurement.
      items of measures such as the Yale Food Addiction Scale (YFAS)
      • Gearhardt A.N.
      • Corbin W.R.
      • Brownell K.D.
      Preliminary validation of the Yale Food Addiction Scale.
      ,
      • Gearhardt A.N.
      • Corbin W.R.
      • Brownell K.D.
      Development of the Yale Food Addiction Scale Version 2.0.
      which is used in the study by Parnarouskis and colleagues,
      • Parnarouskis L.
      • Gearhardt A.N.
      • Mason S.E.
      • et al.
      Association of food insecurity and food addiction symptoms: a secondary analysis of two samples of low-income female adults.
      may be interpreted differently by individuals experiencing food insecurity. As examples, the YFAS includes items that measure eating beyond hunger and eating until feeling physically ill. Yet, individuals with limited resources have identified overeating when food is available as a coping strategy to make it through the month with enough food.
      • Kempson K.
      • Keenan D.P.
      • Sadani P.S.
      • Adler A.
      Maintaining food sufficiency: coping strategies identified by limited-resource individuals versus nutrition educators.
      Thus, in populations with food insecurity, items on the YFAS, which assess overeating-related phenomena may capture strategies to cope with food insecurity rather than symptoms of food addiction. Likewise, the YFAS asks about strong urges to eat certain foods, which, in the context of food insecurity, may stem from deprivation rather than food addiction. In addition, the YFAS item about distress caused by one’s eating behaviors may similarly tap into unique experiences of populations with food insecurity and could thus be interpreted differently in the case that people with food insecurity have a different definition or source of distress than those without food insecurity. It is possible that for people with food insecurity, unplanned eating of even small amounts of food could cause less food to be available for themselves or for their families, which could cause high distress but may not reflect food addiction.
      Considering the reasons that eating behavior measures such as the YFAS could conceivably capture different constructs in people with food insecurity, one must be cautious in interpreting findings using these measures to avoid overpathologizing or misclassifying behaviors. Thus, to advance understanding of maladaptive eating behaviors in these populations and enhance confidence in findings, more psychometric testing of measures is required in populations with food insecurity. This can be achieved using both qualitative and quantitative approaches, as described below.

      Implementing Qualitative Research to Understand Measure Performance

      A critical step toward understanding whether or not a measure appropriately captures the construct of interest across different populations is to assess the extent to which item interpretation is similar across those populations. Qualitative research serves as a valuable tool to achieve this goal. Qualitative studies using interview or focus group procedures can be implemented to evaluate how specific items are interpreted, which situations or contexts respondents consider when responding to questions, and in the case that there are points of confusion regarding how items are worded. A specific approach that is particularly valuable in this realm is cognitive interviewing, which involves gathering information about how individuals arrive at their responses on particular items.
      • Beatty P.C.
      • Willis G.B.
      Research synthesis: the practice of cognitive interviewing.
      In general, qualitative approaches could verify whether or not the items are interpreted and answered in a way that is consistent with the overall construct of interest. For example, Meza and colleagues
      • Meza A.
      • Altman E.
      • Martinez S.
      • Leung C.W.
      “It’s a feeling that one is not worth food”: a qualitative study exploring the psychosocial experience and academic consequences of food insecurity among college students.
      found, using a qualitative approach, that some undergraduate students with food insecurity experienced frequent stressful thoughts about food that interfered with daily life, a sense of not feeling worthy of food, and/or not feeling worthy of help from others in obtaining food. Thus, qualitative research has the ability to help us make inferences about the more nuanced emotional context of reported experiences, such as thinking a lot about food or restricting food.

      Implementing Quantitative Research to Understand Measure Performance

      Future research should also employ specific quantitative approaches to evaluate whether or not eating-related measures perform similarly in individuals with food insecurity compared with individuals without food insecurity. Quantitative approaches can provide a statistical gauge on the extent to which measures assess their intended constructs when administered in individuals with food insecurity. Such approaches could inform particular items or overarching constructs that may need to be interpreted with greater caution or amended to accurately assess/reflect the experiences of individuals with food insecurity.
      First, it is essential to ensure the structural similarity, and thus construct validity, of eating-related measures across samples with and without food insecurity. An initial step could be to use confirmatory factor analysis to verify that measures such as the YFAS replicate on a structural level in individuals with food insecurity. A measurement invariance approach would be ideal for exploring whether or not a measure captures the same constructs in groups with and without food insecurity and whether or not the comparison of scores on a given measure is appropriate. In measurement invariance testing, multigroup confirmatory factor analysis is first conducted to assess whether or not a measure’s factors (usually representing scales or subscales on a measure) replicate across groups of interest (eg, across a group of individuals with food insecurity and a group of individuals without). Next, more restrictive confirmatory factor analysis models are conducted to evaluate whether or not model fit to the data substantially worsens when item factor loadings and item intercepts/thresholds are specified as equal across groups of interest.
      • Van de Schoot R.
      • Lugtig P.
      • Hox J.
      A checklist for testing measurement invariance.
      ,
      • Putnick D.L.
      • Bornstein M.H.
      Measurement invariance conventions and reporting: the state of the art and future directions for psychological research.
      Worsening of model fit with the addition of such constraints (ie, evidence of measurement noninvariance) indicates that notable differences across groups are present at the level being evaluated (eg, at the loading or intercept/threshold level), which means greater caution may be warranted when comparing measure scores across groups. For example, Perez and colleagues
      • Perez M.
      • Ohrt T.K.
      • Bruening A.B.
      • et al.
      Measurement equivalence of child feeding and eating measures across gender, ethnicity, and household food security.
      utilized a measurement invariance approach with the Child Feeding Questionnaire and the Child Eating Behaviour Questionnaire, finding that several constructs varied as a function of food insecurity, leading to the inference that certain constructs were less relevant to households with food insecurity.
      As previously mentioned, there may be conceptual reasons why individuals with food insecurity would respond differently to certain items intended to measure constructs such as food addiction. Future research should thus also consider the extent to which responses on such items may be explained by measurement differences rather than reflecting true group differences in levels of the eating-related construct being measured. Such clarification at the item level can be achieved by testing for differential item functioning. Differential item functioning—which corresponds to measurement noninvariance at the item intercept or factor loading level
      • Lee J.
      • Little T.D.
      • Preacher K.J.
      Methodological issues in using structural equation models for testing differential item functioning.
      —refers to situations in which the probability of endorsing an item differs across groups, even at similar levels of the overall construct being measured.
      • Thissen D.
      • Steinberg L.
      • Wainer H.
      Detection of differential item functioning using the parameters of item response models.
      • Meulders M.
      • Xie Y.
      Person-by-item predictors.
      • Osterlind S.J.
      • Everson H.T.
      Differential Item Functioning.
      Testing for differential item functioning using an item response theory approach may prove particularly informative because it can examine both uniform and nonuniform differential item functioning, as well as yield item characteristic curves depicting the likelihood of endorsing a particular item response at various levels of the underlying latent variable.
      Although measurement invariance testing provides a more comprehensive assessment of how an overall measure performs across groups, differential item functioning may be a useful alternative approach in some circumstances. For example, differential item functioning may be particularly useful for measures whose factor structures do not consistently replicate or whose development was not focused on having a clear, robust factor structure, given that achieving adequate fit in confirmatory factor analysis is a prerequisite for measurement invariance testing. An illustrative example of this approach is provided by O’Connor and colleagues,
      • O’Connor S.M.
      • Hazzard V.M.
      • Zickgraf H.F.
      Exploring differential item functioning on eating disorder measures by food security status.
      who recently detected differential item functioning by food security status on a measure of disordered eating. Specifically, certain items related to overeating on the Eating Disorder Diagnostic Scale demonstrated differential item functioning in individuals with food insecurity, such that incorrect inferences about the severity of disordered eating may be made when interpreting responses on particular items in individuals with food insecurity. This example highlights how differential item functioning can be used to identify certain items that do not perform as measures intended in food-insecure samples.

      Next Steps and Challenges

      Implementing the above qualitative and quantitative approaches is critical to ensure that research on eating-related behaviors in individuals with food insecurity draws valid and reliable conclusions. The consequence of not performing further psychometric validation is that erroneous (and potentially harmful) conclusions may be drawn about the prevalence and severity of different eating-related behaviors, which could result in the misidentification of pathology and/or a failure to identify true concerns facing communities, families, and individuals.
      The work of psychometric validation in populations with food insecurity will also need to consider the heterogeneity of identities represented within this population. Food insecurity often coexists within the complex intersection of variables such as race, ethnicity, socioeconomic class, gender, sexual orientation, and disability. Many people with food insecurity have multiple marginalized identities because food insecurity disproportionately influences Black, Hispanic, and Indigenous communities due to past and present factors such as socioeconomic conditions, structural racism, discrimination, and disruptions to traditional subsistence methods.
      • Fitzpatrick K.M.
      • Harris C.
      • Drawve G.
      • Willis D.E.
      Assessing food insecurity among US adults during the COVID-19 pandemic.
      • Bruening M.
      • MacLehose R.
      • Loth K.
      • Story M.
      • Neumark-Sztainer D.
      Feeding a family in a recession: food insecurity among Minnesota parents.
      • Hernandez D.C.
      • Reesor L.M.
      • Murillo R.
      Food insecurity and adult overweight/obesity: gender and race/ethnic disparities.
      • Wolfson J.A.
      • Leung C.W.
      Food insecurity and COVID-19: disparities in early effects for US adults.
      • Odoms-Young A.M.
      Examining the impact of structural racism on food insecurity: implications for addressing racial/ethnic disparities.
      • Myers C.A.
      • Mire E.F.
      • Katzmarzyk P.T.
      Trends in adiposity and food insecurity among US adults.
      • Morales D.X.
      • Morales S.A.
      • Beltran T.F.
      Racial/ethnic disparities in household food insecurity during the COVID-19 pandemic: a nationally representative study.
      • Patterson J.G.
      • Russomanno J.
      • Teferra A.A.
      • Jabson Tree J.M.
      Disparities in food insecurity at the intersection of race and sexual orientation: a population-based study of adult women in the United States.
      Consequently, psychometric validation must also take into account intersectional identities (which are often not centered in the original validation studies) in ensuring representation within samples and subsamples. Such careful validation work is critical to scientific rigor and is necessary for research on food insecurity to continue to move forward.
      In addition to validation work, researchers should also continue to follow best practices for reporting statistics related to the reliability and validity of measures in their study samples and note when measures are being used with populations not centered in the original validation studies. Furthermore, authors should clearly state the limitations of using such measures in their discussion sections and consider alternate explanations resulting from measurement error. Reliability and validity statistics may aid readers in understanding the performance of different measures and shape their interpretations of findings. It should be noted that common reliability statistics, such as Cronbach alpha, do not establish the validity of a construct in different populations. Furthermore, a measure may be reliable, but not valid, particularly in the case that items are interpreted in a manner that is consistent but does not reflect the intended construct. Thus, such reporting should be considered a minimum—but not sufficient—requirement moving forward.

      Conclusions

      As research on food insecurity grows, we urge researchers studying food insecurity and eating-related behaviors to work together to complete psychometric validation studies, which will enhance the field’s ability to understand phenomena related to food insecurity and to draw conclusions that can positively influence public health, the field of nutrition and dietetics, public policy, and mental health treatment.

      References

        • Parnarouskis L.
        • Gearhardt A.N.
        • Mason S.E.
        • et al.
        Association of food insecurity and food addiction symptoms: a secondary analysis of two samples of low-income female adults.
        J Acad Nutr Diet. 2022; 122: 1885-1892https://doi.org/10.1016/j.jand.2022.04.015
        • Furr M.
        Scale Construction and Psychometrics for Social and Personality Psychology.
        SAGE Publications Ltd, 2011
        • Anderson S.A.
        Core indicators of nutritional state for difficult-to-sample populations.
        J Nutr. 1990; 120: 1555-1600
        • Pinstrup-Andersen P.
        Food security: definition and measurement.
        Food Secur. 2009; 1: 5-7https://doi.org/10.1007/s12571-008-0002-y
        • Gearhardt A.N.
        • Corbin W.R.
        • Brownell K.D.
        Preliminary validation of the Yale Food Addiction Scale.
        Appetite. 2009; 52: 430-436https://doi.org/10.1016/j.appet.2008.12.003
        • Gearhardt A.N.
        • Corbin W.R.
        • Brownell K.D.
        Development of the Yale Food Addiction Scale Version 2.0.
        Psychol Addict Behav. 2016; 30: 113-121https://doi.org/10.1037/adb0000136
        • Kempson K.
        • Keenan D.P.
        • Sadani P.S.
        • Adler A.
        Maintaining food sufficiency: coping strategies identified by limited-resource individuals versus nutrition educators.
        J Nutr Educ Behav. 2003; 35: 179-188
        • Beatty P.C.
        • Willis G.B.
        Research synthesis: the practice of cognitive interviewing.
        Public Opin Q. 2007; 71: 287-311
        • Meza A.
        • Altman E.
        • Martinez S.
        • Leung C.W.
        “It’s a feeling that one is not worth food”: a qualitative study exploring the psychosocial experience and academic consequences of food insecurity among college students.
        J Acad Nutr Diet. 2019; 119: 1713-1721.e1https://doi.org/10.1016/j.jand.2018.09.006
        • Van de Schoot R.
        • Lugtig P.
        • Hox J.
        A checklist for testing measurement invariance.
        Eur J Dev Psychol. 2012; 9: 486-492
        • Putnick D.L.
        • Bornstein M.H.
        Measurement invariance conventions and reporting: the state of the art and future directions for psychological research.
        Dev Rev. 2016; 41: 71-90
        • Perez M.
        • Ohrt T.K.
        • Bruening A.B.
        • et al.
        Measurement equivalence of child feeding and eating measures across gender, ethnicity, and household food security.
        BMC Obes. 2018; 5: 17https://doi.org/10.1186/s40608-018-0192-6
        • Lee J.
        • Little T.D.
        • Preacher K.J.
        Methodological issues in using structural equation models for testing differential item functioning.
        in: Cross-Cultural Analysis. Routledge, 2018: 65-94
        • Thissen D.
        • Steinberg L.
        • Wainer H.
        Detection of differential item functioning using the parameters of item response models.
        in: Holland P.W. Wainer H. Differential Item Functioning. Lawrence Erlbaum Associates, Inc, 1993: 67-113
        • Meulders M.
        • Xie Y.
        Person-by-item predictors.
        in: Explanatory Item Response Models. Springer, 2004: 213-240
        • Osterlind S.J.
        • Everson H.T.
        Differential Item Functioning.
        SAGE Publications, 2009
        • O’Connor S.M.
        • Hazzard V.M.
        • Zickgraf H.F.
        Exploring differential item functioning on eating disorder measures by food security status.
        Eat Weight Disord. 2022; 27: 1449-1455https://doi.org/10.1007/s40519-021-01289-z
        • Fitzpatrick K.M.
        • Harris C.
        • Drawve G.
        • Willis D.E.
        Assessing food insecurity among US adults during the COVID-19 pandemic.
        J Hunger Environ Nutr. 2021; 16: 1-18https://doi.org/10.1080/19320248.2020.1830221
        • Bruening M.
        • MacLehose R.
        • Loth K.
        • Story M.
        • Neumark-Sztainer D.
        Feeding a family in a recession: food insecurity among Minnesota parents.
        Am J Public Health. 2012; 102: 520-526https://doi.org/10.2105/AJPH.2011.300390
        • Hernandez D.C.
        • Reesor L.M.
        • Murillo R.
        Food insecurity and adult overweight/obesity: gender and race/ethnic disparities.
        Appetite. 2017; 117: 373-378https://doi.org/10.1016/j.appet.2017.07.010
        • Wolfson J.A.
        • Leung C.W.
        Food insecurity and COVID-19: disparities in early effects for US adults.
        Nutrients. 2020; 12: 1648https://doi.org/10.3390/nu12061648
        • Odoms-Young A.M.
        Examining the impact of structural racism on food insecurity: implications for addressing racial/ethnic disparities.
        Fam Community Health. 2018; 41: S3-S6https://doi.org/10.1097/FCH.0000000000000183
        • Myers C.A.
        • Mire E.F.
        • Katzmarzyk P.T.
        Trends in adiposity and food insecurity among US adults.
        JAMA Netw Open. 2020; 3 (e2012767-e2012767)https://doi.org/10.1001/jamanetworkopen.2020.12767
        • Morales D.X.
        • Morales S.A.
        • Beltran T.F.
        Racial/ethnic disparities in household food insecurity during the COVID-19 pandemic: a nationally representative study.
        J Racial Ethn Health Disparities. 2021; 8: 1300-1314https://doi.org/10.1007/s40615-020-00892-7
        • Patterson J.G.
        • Russomanno J.
        • Teferra A.A.
        • Jabson Tree J.M.
        Disparities in food insecurity at the intersection of race and sexual orientation: a population-based study of adult women in the United States.
        SSM Popul Health. 2020; 12: 100655https://doi.org/10.1016/j.ssmph.2020.100655

      Biography

      K. A. Christensen is a postdoctoral fellow, Department of Psychology, University of Kansas, Lawrence.

      Biography

      V. M. Hazzard is a postdoctoral fellow, Department of Psychiatry and Behavioral Sciences, University of Minnesota Medical School, Minneapolis, and Division of Epidemiology and Community Health, University of Minnesota School of Public Health, Minneapolis.

      Biography

      B. N. Richson is a graduate student, Department of Psychology, University of Kansas, Lawrence.

      Biography

      K. E. Hagan is a postdoctoral fellow, Department of Psychiatry, Columbia University Irving Medical Center and New York State Psychiatric Institute, New York City.

      Linked Article