Cancer prevention recommendations.
Concepts Of Validity, Reliability, and Validation
THE Exposures of Interest in Dietary Assessment
Self-Report Methods for Assessing Dietary Intake
National Health and Nutrition Examination Survey.
- Amoutzopoulos B.
- Steer T.
- Roberts C.
- et al.
Objective Measures of Dietary Intake
Sources of Error in Data Collected Using Self-Report Methods
Limitations in the Current Dietary Assessment Validation Literature
Best Practices in Dietary Assessment Validation
|Title and abstract||Indicate the study’s design and purpose, including the construct of interest (ie, dietary components of interest and over what time frame) and dietary assessment measure undergoing validation, in the title and/or abstract.|
Provide an informative and balanced summary of what was done and what was found. Avoid summary statements that do not reflect the totality of the findings and that treat validity and/or reliability as dichotomous rather than properties operating on a continuum of low to high.
|Background and rationale||Explain the scientific background and rationale for the validation study. Place the study in the context of existing research on dietary assessment methods, and build justification for the specific focus of the validation study.|
|Objectives||State specific objectives, including any prespecified hypotheses. Objectives should explicitly identify the aim of the study in terms of the properties assessed to examine suitability of the measure to address a given research purpose in a specified population and setting.|
|Study design||Present key elements of the study design early.|
|Measure||Describe the measure undergoing validation in detail, including how it was developed/adapted, its format, the method and location of administration, and salient characteristics such as the associated food composition database(s) and whether supplement intake is assessed.|
Describe the intended use of the measure (ie, to capture occurrence of consumption (or not), to rank intake among a group of persons, or to estimate absolute intake), as well as the dietary components (foods, food groups, nutrients, patterns, or other food components) and time frame of interest.
|Settings||Describe the setting, locations, and relevant dates, including periods of recruitment and data collection. Describe any characteristics of the study setting that might influence participants’ dietary intake.|
|Participants||Provide the eligibility criteria and sources and methods of selection of participants.|
Justify the sample size.
Describe the representativeness of the sample to the target population and discuss response rates. Report characteristics related to nutrition, dietary intake, and physiology considered in defining the eligibility criteria.
|Procedures||Is the measure well-constructed and grounded in an understanding of the underlying phenomenon of interest? (face and content validity)|
|Participants||Report the numbers of individuals at each stage of the study and give reasons for nonparticipation at each stage. Consider the use of a flow diagram to illustrate.|
Report the results of each procedure implemented for each dietary construct of interest.
|Descriptive data||Give information on the study participants (eg, demographic characteristics)|
Provide salient information regarding dietary intake data (eg, low or high values, avoidance of certain foods)
|Key results||Make use of all statistical tests and/or procedures to objectively summarize the key results with reference to the study objectives. Discuss the degree of validity, reliability, sensitivity to change, and/or equivalence of the evaluated measure as appropriate to the study, rather than referring to these properties as present or absent. When comparing error-prone measures to one another, consider the contribution of correlated error to measures of association. Avoid overstating the level of validity or reliability based on the available data.|
|Limitations||Describe study limitations that may affect conclusions. This may include the reference measures in studies of validity or recruitment methods (eg, paid volunteers) in any study.|
|Interpretation||Limit interpretations about validity, reliability, responsiveness, and/or equivalence of the evaluated measure to the specific populations and contexts evaluated, as well as the particular objectives (eg, interpretations of an evaluation of validity should be limited to validity and not reliability). Base interpretations on the totality of the evidence, including all tests and comparisons conducted, as well as results from similar studies. Place the findings in the context of other literature.|
|Generalizability||Describe potential appropriate and inappropriate uses of the measure given the study design and findings. Describe features of the measure that may influence the design of studies proposing to make use of it; for example, sample size calculations to account for loss of power due to biased measurement of dietary intake.|
|Funding||Give the source of funding and the role of the funders in the validation study and, when applicable, in prior studies on which the present validation study is based.|
|Ethics||Describe the procedures for consent and study approval from ethics committee(s).|
Is the Measure Well Constructed and Grounded in an Understanding of the Underlying Phenomenon of Interest?
Does the Measure Perform in a Manner Consistent with the Theory Underlying its Construction?
Is the Measure Accurate within Specified Performance Standards?
Studies Comparing a Method to an Unbiased Reference Measure
Studies Comparing a Method to a Biased or Error-Prone Reference Measure
Does the Measure Produce Data that Are Precise and Dependable?
Is the Measure Responsive to Change?
Does the Measure Produce Data that Are Equivalent or Comparable across Populations?
Interpreting and Reporting the Findings of Validation Studies
- 1.Inferences should be limited to the specific objectives of the validation study, and the specific populations and contexts in which it was conducted, in addition to differentiating findings across dietary components. Blanket statements regarding validity and reliability should be avoided. For example, it is not appropriate to conclude that a measure is valid and reliable based on a study focused only on reliability. Likewise, it is not appropriate to conclude that a measure has high validity for measuring dietary intake when findings differ according to different dietary components.
- 2.Validity is “viewed as a carefully structured argument assembling evidence from a variety of sources to support or refute proposed interpretations” of data from a method.31Thus, the totality of evidence, including all tests and comparisons conducted, should be weighed in relation to the purpose of the measure and of the validation study, as well as the study’s strengths and limitations. This includes considerations regarding the reference measure used to assess validity. For example, in the case that a biased comparison measure is used to assess validity, the only conclusion that can be reached is the extent to which the data collected correspond to that collected using the reference, not the extent to which the evaluated tool captures accurate data. When an unbiased criterion measure is used, stronger conclusions about the potential value of the tool are warranted, subject to caveats related to the study’s other strengths and weaknesses. Kelly and colleagues34refer to purpose and context validity, referring to whether all assessments conducted indicate that the measure is “suitable for the proposed use and likely to allow the research question to be answered” and the extent to which the measure will provide useful information given the proposed context.
- 3.Statistical tests should be interpreted with attention to not only statistical significance but also the meaningfulness of the results. Although cut-points, for example, for correlation coefficients and κ values have been proposed, their application to determine that a tool is valid or reliable should be used cautiously with consideration of the totality of the evidence and the intended uses of the measure. For example, for studies that assess criterion validity, attenuation and correlation values below 0.4 (reflective of estimation of a true relative risk of 2 as 1.32, in the context of a diet–disease study) have been viewed as undesirable, but this is not a sharp cut-point.85In addition, with large samples, correlations may be statistically significant, but have no practical significance.
- 4.Inferences should be nuanced, recognizing that constructs such as validity and reliability operate on continuums from low to high, and degrees in between may be appropriate depending on the research objective. For example, a dietary assessment measure may capture intake accurately enough to allow differentiation of high from low consumers, but not to compare intake with sufficient accuracy for comparison to nutrient requirements or food group recommendations.
- 5.Inferences should reflect the characteristics of the measure validated. Although a particular iteration of a 24HR or FR may have been shown to have high validity for capturing intake of a given dietary construct in a given population over a specific period of time, this may not be true for other variations of the method; for example, using different modes of administration.
- 6.Findings should be placed in the context of those from similar studies to assess the potential value and uses of the method compared with other available methods. The links between different systems of validity provided here and in Frongillo and colleagues29are intended to ensure that terms are clearly explained and related to other terms that might be used in articles reporting on other validation studies. This should support appropriate interpretation and synthesis of the literature.
- 7.Based on the study design and findings, authors should describe potentially appropriate and inappropriate uses of the measure and features that may influence the design of studies proposing to use it. For example, methods that capture diet with substantial random error will generally lead to attenuation of associations within epidemiologic research. Thus, such tools may not be useful for this purpose unless this error can be mitigated (eg, through the collection of repeat measures and statistical modelling to adjust for the random error26). In addition, findings from validation studies can be useful for estimating the loss of power due to measurement error (as error increases, the required sample size to achieve the same level of statistical power also increases).111Authors should also be clear about what their findings do not imply. For example, as noted, a comparison of two error-prone measures cannot be used to conclude that a method is valid but rather how it compares with the measure that served as the reference.
- 8.Researchers making use of the literature to select and justify a given measure should critically evaluate the available validation studies and clearly and transparently convey the findings in subsequent publications. For example, stating that a method has been shown to be valid or previously validated is insufficient for readers to assess whether the method is suitable for the given purpose and context.
- Supplementary Materials
- The dietary history as a tool in research.J Am Diet Assoc. 1947; 23: 1041-1046
- Health effects of dietary risks in 195 countries, 1990–2017: A systematic analysis for the Global Burden of Disease Study 2017.Lancet. 2019; 393: 1958-1972
- Cancer prevention recommendations.(Accessed)
- INFORMAS (International network for food and obesity/non-communicable diseases research, monitoring and action support): Summary and future directions.Obes Rev. 2013; 14: 157-164
- Towards global benchmarking of food environments and policies to reduce obesity and diet-related non-communicable diseases: Design and methods for nation-wide surveys.BMJ Open. 2014; 4: e005339
- A food policy package for healthy diets and the prevention of obesity and diet-related non-communicable diseases: The NOURISHING framework.Obes Rev. 2013; 14: 159-168
- Opinion: A wolf in sheep’s clothing.The Scientist. 2013; (https://www.the-scientist.com/opinion/opinion-a-wolf-in-sheeps-clothing-38534. Published October 22, 2013. Accessed August 8, 2019.)
- Controversy and debate: Memory-based methods paper 1: The fatal flaws of food frequency questionnaires and other memory-based dietary assessment methods.J Clin Epidemiol. 2018; 104: 113-124
- Pathways and mechanisms linking dietary components to cardiometabolic disease: Thinking beyond calories.Obes Rev. 2018; 19: 1205-1235
- Extending methods in dietary patterns research.Nutrients. 2018; 10: 571
- Nutritional Anthropology: Contemporary Approaches to Diet and Culture.Redgrave Publishing, New York, NY1980
- Sociology on the Menu.Routledge, London, UK1996
- Americans do not meet federal dietary recommendations.J Nutr. 2010; 140: 1832-1838
- Income and race/ethnicity are associated with adherence to food-based dietary guidance among US adults and children.J Acad Nutr Diet. 2012; 112: 624-635
- Prioritization of pesticides based on daily dietary exposure potential as determined from the SHEDS model.Food Chem Toxicol. 2016; 96: 167-173
- Energy density of diets reported by American adults: Association with food group intake, nutrient intake and body weight.Int J Obes (Lond). 2005; 29: 950-956
- Examining dietary patterns in relation to chronic disease: Matching measures and methods to questions of interest.Circulation. 2015; 132: 790-793
- Dietary patterns: Challenges and opportunities in dietary patterns research.J Acad Nutr Diet. 2007; 107: 1233-1239
- Evaluation of methodologies for assessing the overall diet: Dietary quality scores and dietary pattern analysis.Proc Nutr Soc. 2013; 72: 191-199
- Secular trends in patterns of self-reported food consumption of adult Americans: NHANES 1971-1975 to NHANES 1999–2002.Am J Clin Nutr. 2006; 84: 1215-1223
- Snacking increased among U.S. adults between 1977 and 2006.J Nutr. 2010; 140: 325-332
- Number of days of food intake records required to estimate individual and group nutrient intakes with defined confidence.J Nutr. 1987; 117: 1638-1641
- Statistical estimation of dietary parameters: Implications of patterns in within-subject variation—a case study of sampling strategies.Am J Clin Nutr. 1992; 55: 22-27
- Temporal dietary patterns derived among the adult participants of the National Health and Nutrition Examination Survey 1999-2004 are associated with diet quality.J Acad Nutr Diet. 2016; 116: 283-291
- The nature and individuality of within-subject variation in energy intake.Am J Clin Nutr. 1991; 54: 464-470
- Statistical methods for estimating usual intake of nutrients and foods: A review of the theory.J Am Diet Assoc. 2006; 106: 1640-1650
- Dietary Assessment Methodology.in: Coulston A.M. Boushey C.K. Ferruzzi M.G. Delahanty L.M. Nutrition in the Prevention and Treatment of Disease. 4th edition. Academic Press, 2017: 5-48
- The National Cancer Institute’s dietary assessment primer: A resource for diet research.J Acad Nutr Diet. 2015; 115: 1986-1995
Frongillo EA, Baranowski T, Subar AF, Tooze JA, Kirkpatrick SI. Establishing validity and cross-context equivalence of measures and indicators. J Acad Nutr Diet. 2018 November 20. Epub ahead of print.
- Application and interpretation of multiple statistical tests to evaluate validity of dietary intake assessment methods.Nutr J. 2015; 14: 40
- Current concepts in validity and reliability for psychometric instruments: Theory and application.Am J Med. 2006; 119 (166.e7-166.e16)
Habicht J-P, Yarbrough C, Martorell R. Anthropometric field methods: Criteria for selection. In: Jelliffe DB, Jelliffe EFP, eds. Nutrition and Growth (A Comprehensive Treatise). Vol 2. Boston, MA: Springer; 1979:365-387.
- Publishing nutrition research: Validity, reliability, and diagnostic test assessment in nutrition-related research.J Am Diet Assoc. 2010; 110: 409-419
- Should we reframe how we think about physical activity and sedentary behaviour measurement? Validity and reliability reconsidered.Int J Behav Nutr Phys Act. 2016; 13: 32
- A psychometric toolbox for testing validity and reliability.J Nurs Scholarsh. 2007; 39: 155-164
- So you want to develop a survey: Practical recommendations for scale development.Am J Health Stud. 2011; 26: 97-105
- Validity and reliability reporting practices in the field of health education and behavior: A review of seven journals.Health Educ Behav. 2014; 41: 12-18
- Dietary assessment primer.
- Quantifying test–retest reliability using the intraclass correlation coefficient and the SEM.J Strength Cond Res. 2005; 19: 231-240
- 2015-2020 Dietary Guidelines for Americans.8th edition. US Government Printing Office, Washington, DC2015
- Trends in blood mercury concentrations and fish consumption among U.S. women of reproductive age, NHANES, 1999–2010.Environ Res. 2014; 133: 431-438
- A new statistical method for estimating the usual intake of episodically consumed foods with application to their distribution.J Am Diet Assoc. 2006; 106: 1575-1587
- Dietary intake patterns are consistent across seasons in a cohort of healthy adults in a metropolitan population.J Acad Nutr Diet. 2016; 116: 38-45
- The developmental origins of adult disease.J Am Coll Nutr. 2004; 23: 588S-595S
- Fetal origins of coronary heart disease.BMJ. 1995; 311: 171-174
- A statistical model for measurement error that incorporates variation over time in the target measure, with application to nutritional epidemiology.Stat Med. 2015; 34: 3590-3605
- Survival analysis with error-prone time-varying covariates: A risk set calibration approach.Biometrics. 2011; 67: 50-58
- Taking advantage of the strengths of 2 different dietary assessment instruments to improve intake estimates for nutritional epidemiology.Am J Epidemiol. 2012; 175: 340-347
- Combining a food frequency questionnaire with 24-hour recalls to increase the precision of estimation of usual dietary intakes—Evidence from the validation studies pooling project.Am J Epidemiol. 2018; 187: 2227-2232
- Measurement error webinar series.
- National Health and Nutrition Examination Survey.
- The Automated Self-Administered 24-hour dietary recall (ASA24): A resource for researchers, clinicians, and educators from the National Cancer Institute.J Acad Nutr Diet. 2012; 112: 1134-1137
- Image-assisted dietary assessment: A systematic review of the evidence.J Acad Nutr Diet. 2015; 115: 64-77
- An overview of the Technology Assisted Dietary Assessment project at Purdue University.ISM. 2010; : 290-295
- Development of a UK online 24-h dietary assessment tool: myfood24.Nutrients. 2015; 7: 4016-4032
- Traditional methods v. new technologies— dilemmas for dietary assessment in large-scale nutrition surveys and studies: A report following an international panel discussion at the 9th International Conference on Diet and Activity Methods (ICDAM9), Brisbane, 3 September 2015.J Nutr Sci. 2018; 7: e11
- The US Department of Agriculture automated multiple-pass method reduces bias in the collection of energy intakes.Am J Clin Nutr. 2008; 88: 324-332
- The USDA Automated Multiple-Pass Method accurately estimates group total energy and nutrient intake.J Nutr. 2006; 136: 2594-2599
- The USDA Automated Multiple-Pass Method accurately assesses population sodium intakes.Am J Clin Nutr. 2013; 97: 958-964
- The use of digital images in 24-hour recalls may lead to less misestimation of portion size compared with traditional interviewer-administered recalls.J Nutr. 2016; 146: 2567-2573
- Inventory of experiences from national/regional dietary monitoring surveys using EPIC-Soft.Eur J Clin Nutr. 2011; 65: S16-S28
- Structure of dietary measurement error: Results of the OPEN biomarker study.Am J Epidemiol. 2003; 158: 14-21
- Warm D. Development, validation and utilisation of food-frequency questionnaires—a review.Public Health Nutr. 2002; 5: 567-587
- Performance of the Automated Self-Administered 24-hour Recall relative to a measure of true intakes and to an interviewer-administered 24-h recall.Am J Clin Nutr. 2014; 100: 233-240
- Interview format influences the accuracy of children’s dietary recalls validated with observations.Nutr Res. 2003; 23: 1537-1546
- Uses and limitations of statistical accounting for random error correlations, in the validation of dietary questionnaire assessments.Public Health Nutr. 2002; 5: 969-976
- Addressing current criticism regarding the value of self-report dietary data.J Nutr. 2015; 145: 2639-2645
- Reliability of the doubly labeled water method for the measurement of total daily energy expenditure in free-living subjects.J Nutr. 1996; 126: 348S-354S
- Urine nitrogen as a biomarker for the validation of dietary protein intake.J Nutr. 2003; 133: 921S-924S
- Pooled results from 5 validation studies of dietary self-report instruments using recovery biomarkers for potassium and sodium intake.Am J Epidemiol. 2015; 181: 473-487
- Urinary sugars—a biomarker of total sugars intake.Nutrients. 2015; 7: 5816-5833
- Use of the predictive sugars biomarker to evaluate self-reported total sugars intake in the Observing Protein and Energy Nutrition (OPEN) study.Cancer Epidemiol Biomarkers Prev. 2011; 20: 490-500
- Urinary sucrose and fructose as biomarkers for sugar consumption.Cancer Epidemiol Biomarkers Prev. 2005; 14: 1287-1294
- Biomarkers in nutritional epidemiology: Applications, needs and new horizons.Hum Genet. 2009; 125: 507-525
- Dietary biomarker evaluation in a controlled feeding study in women from the Women’s Health Initiative cohort.Am J Clin Nutr. 2016; 105: 466-475
- Validation of dietary assessment methods in the UK arm of EPIC using weighed records, and 24-hour urinary nitrogen and potassium and serum vitamin C and carotenoids as biomarkers.Int J Epidemiol. 1997; 26: S137
- Validation of overweight children's fruit and vegetable intake using plasma carotenoids.Obesity. 2009; 17: 162-168
- Measurement Error in Nonlinear Models: A Modern Perspective.Chapman and Hall/CRC, New York, NY2006
- Random measurement error and regression dilution bias.BMJ. 2010; 340: c2289
- Reactivity and its association with body mass index across days on food checklists.J Acad Nutr Diet. 2012; 112: 110-118
- Dietary assessment methods in epidemiological research: Current state of the art and future prospects.F1000Res. 2017; 6: 926
- Effects of social approval bias on self-reported fruit and vegetable consumption: A randomized controlled trial.Nutr J. 2008; 7: 18
- Social desirability bias in self-reported dietary, physical activity and weight concerns measures in 8- to 10-year-old African-American girls: Results from the Girls Health Enrichment Multisite Studies (GEMS).Prev Med. 2004; 38: 78-87
- Social desirability bias in dietary self-report may compromise the validity of dietary intake measures.Int J Epidemiol. 1995; 24: 389-398
- Pooled results from 5 validation studies of dietary self-report instruments using recovery biomarkers for energy and protein intake.Am J Epidemiol. 2014; 180: 172-188
- Strengthening the reporting of observational studies in epidemiology—Nutritional epidemiology (STROBE-nut): An extension of the STROBE statement.PLOS Med. 2016; 13e1002036
- Measurement error of dietary self-report in intervention trials.Am J Epidemiol. 2010; 172: 819-827
- The Delphi technique: Making sense of consensus.PARE. 2007; 12: 1-8
- Evaluation of the Healthy Eating Index-2015.J Acad Nutr Diet. 2018; 118: 1622-1633
- Origins of intrusions in children’s dietary recalls: Data from a validation study concerning retention interval and information from school food-service production records.Public Health Nutr. 2009; 12: 1569-1575
- Intrusions in children’s dietary recalls: The roles of BMI, sex, race, interview protocol, and social desirability.Obesity (Silver Spring). 2008; 16: 2169-2174
- Assessment of the accuracy of portion size reports using computer-based food photographs aids in the development of an automated self-administered 24-hour recall.J Am Diet Assoc. 2010; 110: 55-64
- Statistical methods for assessing agreement between two methods of clinical measurement.Lancet. 1986; 327: 307-310
- Measuring agreement in method comparison studies.Stat Methods Med Res. 1999; 8: 135-160
- Understanding Bland-Altman analysis.Biochem Med. 2015; 25: 141-151
- Between-and within-subject variation in nutrient intake from infancy to old age: Estimating the number of days required to rank dietary intakes with desired precision.Am J Clin Nutr. 1989; 50: 155-167
- Measuring change over time: assessing the usefulness of evaluative instruments.J Chronic Dis. 1987; 40: 171-178
- Review of evaluation tools used to assess the impact of nutrition education on dietary intake and quality, weight management practices, and physical activity of low-income audiences.J Nutr Educ. 2001; 33: S35-S48
- A measure of quality of life for clinical trials in chronic lung disease.Thorax. 1987; 42: 773-778
- Validity, reliability, and sensitivity to change of a French version of the arthritis impact measurement scales 2 (AIMS2) in patients with rheumatoid arthritis treated with methotrexate.J Rheumatol. 1996; 23: 52-60
- Validity, reliability, and responsiveness of the Spanish version of the OPTIMAL instrument.PM R. 2018; 11: 258-269
- Limited responsiveness related to the minimal important difference of patient-reported outcomes in rare diseases.J Clin Epidemiol. 2016; 79: 10-21
- Reliability and sensitivity to change of measurement instruments used in a traumatic brain injury population.Clin Rehabil. 2006; 20: 686-700
- Responsiveness and validity in health status measurement: A clarification.J Clin Epidemiol. 1989; 42: 403-408
- Internal consistency reliability is a poor predictor of responsiveness.Health Qual Life Outcomes. 2005; 3: 33
- Establishing the minimal number of items for a responsive, valid, health-related quality of life instrument.J Clin Epidemiol. 2001; 54: 571-579
- Responsiveness to change of self-report and device-based physical activity measures in the Living Well with Diabetes Trial.J Phys Act Heal. 2015; 12: 1082-1087
- Biomarker responses to folic acid intervention in healthy adults: A meta-analysis of randomized controlled trials.Am J Clin Nutr. 2014; 99: 96-106
- The standardized computerized 24-h dietary recall method EPIC-Soft adapted for pan-European dietary monitoring.Eur J Clin Nutr. 2011; 65: S5-S15
- Diversity of dietary patterns observed in the European Prospective Investigation into Cancer and Nutrition (EPIC) project.Public Health Nutr. 2002; 5: 1311-1328
- Dealing with dietary measurement error in nutritional cohort studies.J Natl Cancer Inst. 2011; 103: 1086-1092
Supplementary materials: PowerPoint presentation available at www.jandonline.org
STATEMENT OF POTENTIAL CONFLICT OF INTEREST No potential conflict of interest was reported by the authors.
FUNDING/SUPPORT There is no funding to disclose.
AUTHOR CONTRIBUTIONS All authors conceptualized the manuscript. S. I. Kirkpatrick led the drafting of the manuscript and all authors contributed critical content and edits.