Academy of Nutrition and Dietetics Nutrition Research Network: Validation of a Novel Nutrition Informatics Tool to Assess Agreement Between Documented Nutrition Care and Evidence-Based Recommendations

Published:April 23, 2021DOI:


      More evidence regarding registered dietitian nutritionist implementation of evidence-based nutrition practice guidelines (EBNPGs) is needed. We assessed the utility of an automated informatics tool to evaluate congruence of documented nutrition care with 13 individual recommendations in the diabetes mellitus (DM) EBNPG and with the guideline overall. A concurrent validation study was conducted using Nutrition Care Process Terminology documentation entered in the Academy of Nutrition and Dietetics Health Informatics Infrastructure by registered dietitian nutritionists caring for patients with DM. A 15% subset (n = 115) of the 790 patient encounters recorded were selected randomly, and the documented care was evaluated using the automated DM Expected Care Plan (ECP) Analyzer and expert audit. Recommendation-level congruence, as determined by each method, was compared using Cohen’s κ analysis, and the accuracy, sensitivity, and specificity of the DM ECP Analyzer for assessing overall guideline-level congruence was calculated with expert audits as the “gold standard.” For recommendation-level congruence, the DM ECP Analyzer identified more instances of recommendation implementation in the patient encounters, and classified more encounters as including partial or full recommendation implementation for 10 of the 13 recommendations, compared with the expert audit. There was slight to fair agreement between the DM ECP and the expert audit for most individual recommendations, with a mean ± standard deviation level of agreement of κ = .17 ± .19 across all eligible recommendations. At the guideline level, the DM Analyzer had high accuracy (98.3%) and sensitivity (99.1%) and low specificity (0%; no true negatives detected). The DM ECP Analyzer is acceptable for conducting automated audits of nutrition documentation to assess congruence of documented care with recommendations for evidence-based care. Future changes to the EBNPG, Nutrition Care Process Terminology, Academy of Nutrition and Dietetics Health Informatics Infrastructure, and the DM ECP Analyzer could potentially improve recommendation-level performance. The DM ECP Analyzer can be modified for other EBNPGs to facilitate automated assessment of guideline implementation.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of the Academy of Nutrition and Dietetics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Gray G.E.
        • Gray L.K.
        Evidence-based medicine: Applications in dietetic practice.
        J Am Diet Assoc. 2002; 102: 1263-1272
        • Handu D.
        • Moloney L.
        • Wolfram T.
        • Ziegler P.
        • Acosta A.
        • Steiber A.
        Academy of Nutrition and Dietetics methodology for conducting systematic Reviews for the Evidence Analysis Library.
        J Acad Nutr Diet. 2016; 116: 311-318
        • Papoutsakis C.
        • Moloney L.
        • Sinley R.C.
        • Acosta A.
        • Handu D.
        • Steiber A.L.
        Academy of Nutrition and Dietetics Methodology for Developing Evidence-Based Nutrition Practice Guidelines.
        J Acad Nutr Diet. 2017; 117: 794-804
        • Franz M.J.
        • MacLeod J.
        • Evert A.
        • et al.
        Academy of Nutrition and Dietetics Nutrition Practice Guideline for type 1 and type 2 diabetes in adults: Nutrition intervention evidence reviews and recommendations.
        J Acad Nutr Diet. 2017; 117: 1637-1658
      1. Academy Mission, Vision and principles. Academy of Nutrition and Dietetics.
        (Updated 2020. Accessed March 5, 2021)
        • Murphy W.J.
        • Hand R.K.
        • Abram J.K.
        • Papoutsakis C.
        Impact of diabetes prevention guideline adoption on health outcomes: A pragmatic implementation trial. J Acad Nutr Diet. Published online December 3, 2020.
        • Kreimeyer K.
        • Foster M.
        • Pandey A.
        • et al.
        Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review.
        J Biomed Inform. 2017; 73: 14-29
        • Sheikhalishahi S.
        • Miotto R.
        • Dudley J.T.
        • Lavelli A.
        • Rinaldi F.
        • Osmani V.
        Natural language processing of clinical notes on chronic diseases: Systematic review.
        JMIR Med Inform. 2019; 7e12239
        • Swan W.I.
        • Pertel D.G.
        • Hotson B.
        • et al.
        Nutrition Care Process (NCP) update part 2: Developing and using the NCP terminology to demonstrate efficacy of nutrition care and related outcomes.
        J Acad Nutr Diet. 2019 May; 119: 840-855
        • Swan W.I.
        • Vivanti A.
        • Hakel-Smith N.A.
        • et al.
        Nutrition Care Process and Model update: Toward realizing people-centered care and outcomes management.
        J Acad Nutr Diet. 2017; 117: 2003-2014
        • Garner J.A.
        • Proaño G.V.
        • Kelley K.
        • Banna J.C.
        • Emenaker N.J.
        • Sauer K.
        Revising the Academy’s research priorities: Methods of the Research Priorities and Strategies Development Task Force, 2017-2019. J Acad Nutr Diet. Published online November 20, 2020.
        • Streiner D.
        • Norman G.
        Health Measurement Scales: A Practical Guide to Their Development and Use.
        2nd ed. Oxford University Press, 1996
        • Bannigan K.
        • Watson R.
        Reliability and validity in a nutshell.
        J Clin Nurs. 2009; 18: 3237-3243
        • Piedmont R.L.
        Criterion validity.
        in: Michalos A.C. Encyclopedia of Quality of Life and Well-Being Research. Springer Netherlands, 2014: 1348
        • Murphy W.J.
        • Yadrick M.M.
        • Steiber A.L.
        • Mohan V.
        • Papoutsakis C.
        Academy of Nutrition and Dietetics Health Informatics Infrastructure (ANDHII): A pilot study on the documentation of the Nutrition Care Process and the usability of ANDHII by registered dietitian nutritionists.
        J Acad Nutr Diet. 2018; 118: 1966-1974
        • Nutrition Research Network Projects
        Academy of Nutrition and Dietetics.
        (Published 2020. Accessed March 5, 2021)
      2. Coded Private Information or Specimens Use in Research, Guidance. US Department of Health and Human Services. Office for Human Research Protections.
        (Updated 2008. Accessed March 5, 2021)
        • Hakel-Smith N.
        • Lewis N.M.
        A standardized nutrition care process and language are essential components of a conceptual model to guide and document nutrition care and patient outcomes.
        J Am Diet Assoc. 2004; 104: 1878-1884
        • Thompson K.L.
        • Davidson P.
        • Swan W.I.
        • et al.
        Nutrition care process chains: The "missing link" between research and evidence-based practice.
        J Acad Nutr Diet. 2015; 115: 1491-1498
        • Khan A.
        • Baharudin B.
        • Lee L.H.
        • Khan K.
        A review of machine learning algorithms for text-documents classification.
        J Adv Inform Technol. 2010; 1: 4-20
        • Lang C.
        • Siemens G.
        • Wise A.
        • Gašević D.
        Handbook of Learning Analytics.
        1st ed. Society for Learning Analytics Research, 2017
      3. OILS Twitter Scraper [computer program]. Creative Commons Attribution-ShareAlike 4.0 International License, 2014
      4. Microsoft Excel [computer program]. Version 2010. Microsoft, 2010
      5. Stata SE [computer program]. Version 16. StataCorp, 2019
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Jiao Y.
        • Du P.
        Performance measures in evaluating machine learning based bioinformatics predictors for classifications.
        Quant Biol. 2016; : 320-330
      6. Electronic Nutrition Care Process Terminology (eNCPT). NCP step 2: Nutrition diagnosis. Academy of Nutrition and Dietetics.
        (Updated 2020. Accessed March 5, 2021)


      E. Lamers-Johnson is a nutrition researcher, Academy of Nutrition and Dietetics, Nutrition Research Network, Chicago, IL.


      K. Kelley is a nutrition researcher, Academy of Nutrition and Dietetics, Nutrition Research Network, Chicago, IL.


      D. M. Sánchez is an adjunct faculty member, University of New Mexico, Albuquerque.


      K. L. Knippen is an assistant professor, Bowling Green State University, Bowling Green, OH.


      M. Nadelson is an inpatient diabetes educator, Disease Specific Programs, Sarasota Memorial Hospital, Sarasota, FL.


      C. Papoutsakis is a senior director, Data Science Center, Academy of Nutrition and Dietetics, Chicago, IL.


      E. Yakes Jimenez is director, Nutrition Research Network, Academy of Nutrition and Dietetics, Chicago, IL, and a research associate professor, Departments of Pediatrics and Internal Medicine, College of Population Health, University of New Mexico Health Sciences Center, Albuquerque.