Advertisement

Development and Validation of Trigger Algorithms to Identify Delays in Diagnostic Evaluation of Gastroenterological Cancer

  • Daniel R. Murphy
    Correspondence
    Reprint requests Address requests for reprints to: Daniel R. Murphy, MD, MBA, Michael E. DeBakey Veterans Affairs Medical Center, Houston Center for Innovation in Quality, Effectiveness and Safety (152), 2002 Holcombe Boulevard, Houston, Texas 77030. fax: (713) 748-7359.
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
  • Ashley N.D. Meyer
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
  • Viralkumar Vaghani
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
  • Elise Russo
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
  • Dean F. Sittig
    Affiliations
    University of Texas Health Science Center, University of Texas–Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas
    Search for articles by this author
  • Li Wei
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
  • Louis Wu
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
    Search for articles by this author
  • Hardeep Singh
    Affiliations
    Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas

    Department of Medicine, Baylor College of Medicine, Houston, Texas
    Search for articles by this author
Published:August 10, 2017DOI:https://doi.org/10.1016/j.cgh.2017.08.007

      Background & Aims

      Colorectal cancer (CRC) and hepatocellular cancer (HCC) are common causes of death and morbidity, and patients benefit from early detection. However, delays in follow-up of suspicious findings are common, and methods to efficiently detect such delays are needed. We developed, refined, and tested trigger algorithms that identify patients with delayed follow-up evaluation of findings suspicious of CRC or HCC.

      Methods

      We developed and validated two trigger algorithms that detect delays in diagnostic evaluation of CRC and HCC using laboratory, diagnosis, procedure, and referral codes from the Department of Veteran Affairs National Corporate Data Warehouse. The algorithm initially identified patients with positive test results for iron deficiency anemia or fecal immunochemical test (for CRC) and elevated α-fetoprotein results (for HCC). Our algorithm then excluded patients for whom follow-up evaluation was unnecessary, such as patients with a terminal illness or those who had already completed a follow-up evaluation within 60 days. Clinicians reviewed samples of both delayed and nondelayed records, and review data were used to calculate trigger performance.

      Results

      We applied the algorithm for CRC to 245,158 patients seen from January 1, 2013, through December 31, 2013 and identified 1073 patients with delayed follow up. In a review of 400 randomly selected records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 56.0% (95% CI, 51.0%–61.0%). We applied the algorithm for HCC to 333,828 patients seen from January 1, 2011 through December 31, 2014, and identified 130 patients with delayed follow-up. During manual review of all 130 records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 82.3% (95% CI, 74.4%–88.2%). When we extrapolated the findings to all patients with abnormal results, the algorithm identified patients with delayed follow-up evaluation for CRC with 68.6% sensitivity (95% CI, 65.4%–71.6%) and 81.1% specificity (95% CI, 79.5%–82.6%); it identified patients with delayed follow-up evaluation for HCC with 89.1% sensitivity (95% CI, 81.8%–93.8%) and 96.5% specificity (95% CI, 94.8%–97.7%). Compared to nonselective methods, use of the algorithm reduced the number of records required for review to identify a delay by more than 99%.

      Conclusions

      Using data from the Veterans Affairs electronic health record database, we developed an algorithm that greatly reduces the number of record reviews necessary to identify delays in follow-up evaluations for patients with suspected CRC or HCC. This approach offers a more efficient method to identify delayed diagnostic evaluation of gastroenterological cancers.

      Keywords

      Abbreviations used in this paper:

      AFP (α-fetoprotein), CI (confidence interval), CRC (colorectal cancer), EHR (electronic health record), FIT (fecal immunochemical test), FOBT (fecal occult blood test), HCC (hepatocellular cancer), ICD (International Classification of Diseases), IDA (iron deficiency anemia), NPV (negative predictive value), PPV (positive predictive value), VA (Department of Veterans Affairs)
      To read this article in full you will need to make a payment
      AGA Member Login
      Login with your AGA username and password.
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Altekruse S.F.
        • McGlynn K.A.
        • Reichman M.E.
        Hepatocellular carcinoma incidence, mortality, and survival trends in the United States from 1975 to 2005.
        J Clin Oncol. 2009; 27: 1485-1491
        • Mittal S.
        • El-Serag H.B.
        Epidemiology of HCC: consider the population.
        J Clin Gastroenterol. 2013; 47: S2-S6
      1. SEER Cancer Stat Fact Sheets. Available from: http://seer.cancer.gov/statfacts/. Accessed: July 27, 2016.

        • Lin J.S.
        • Piper M.A.
        • Perdue L.A.
        • et al.
        Screening for colorectal cancer: updated evidence report and systematic review for the US Preventive Services Task Force.
        JAMA. 2016; 315: 2576-2594
        • Marquardt J.U.
        • Nguyen-Tat M.
        • Galle P.R.
        • et al.
        Surveillance of hepatocellular carcinoma and diagnostic algorithms in patients with liver cirrhosis.
        Visc Med. 2016; 32: 110-115
        • Singh H.
        • Arora H.S.
        • Vij M.S.
        • et al.
        Communication outcomes of critical imaging results in a computerized notification system.
        J Am Med Inform Assoc. 2007; 14: 459-466
        • Singh H.
        • Petersen L.A.
        • Daci K.
        • et al.
        Reducing referral delays in colorectal cancer diagnosis: is it about how you ask?.
        Qual Saf Health Care. 2010; 19: e27
        • Singh H.
        • Thomas E.J.
        • Sittig D.F.
        • et al.
        Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain?.
        Am J Med. 2010; 123: 238-244
        • Poon E.G.
        • Gandhi T.K.
        • Sequist T.D.
        • et al.
        “I wish I had seen this test result earlier!”: dissatisfaction with test result management systems in primary care.
        Arch Intern Med. 2004; 164: 2223-2228
        • Graber M.L.
        • Franklin N.
        • Gordon R.
        Diagnostic error in internal medicine.
        Arch Intern Med. 2005; 165: 1493-1499
        • Singh H.
        • Giardina T.D.
        • Petersen L.A.
        • et al.
        Exploring situational awareness in diagnostic errors in primary care.
        BMJ Qual Saf. 2012; 21: 30-38
        • Murphy D.R.
        • Singh H.
        • Berlin L.
        Communication breakdowns and diagnostic errors: a radiology perspective.
        Diagnosis. 2014; 1: 253-261
        • Menon S.
        • Smith M.W.
        • Sittig D.F.
        • et al.
        How context affects electronic health record-based test result follow-up: a mixed-methods evaluation.
        BMJ Open. 2014; 4: e005985
        • Singh H.
        • Sethi S.
        • Raber M.
        • et al.
        Errors in cancer diagnosis: current understanding and future directions.
        J Clin Oncol. 2007; 25: 5009-5018
        • Tørring M.L.
        • Frydenberg M.
        • Hansen R.P.
        • et al.
        Time to diagnosis and mortality in colorectal cancer: a cohort study in primary care.
        Br J Cancer. 2011; 104: 934-940
        • Neal R.D.
        • Tharmanathan P.
        • France B.
        • et al.
        Is increased time to diagnosis and treatment in symptomatic cancer associated with poorer outcomes? Systematic review.
        Br J Cancer. 2015; 112: S92-S107
        • Beckman H.B.
        • Markakis K.M.
        • Suchman A.L.
        • et al.
        The doctor-patient relationship and malpractice. Lessons from plaintiff depositions.
        Arch Intern Med. 1994; 154: 1365-1370
        • Singh H.
        • Thomas E.J.
        • Petersen L.A.
        • et al.
        Medical errors involving trainees: a study of closed malpractice claims from 5 insurers.
        Arch Intern Med. 2007; 167: 2030-2036
        • Gale B.D.
        • Bissett-Siegel D.P.
        • Davidson S.J.
        • et al.
        Failure to notify reportable test results: significance in medical malpractice.
        J Am Coll Radiol. 2011; 8: 776-779
        • Singh H.
        • Allen J.I.
        Patient safety counterpoint: systems approaches and multidisciplinary strategies at the centerpiece of error prevention.
        Clin Gastroenterol Hepatol. 2015; 13: 824-826
        • Sittig D.F.
        • Singh H.
        A new sociotechnical model for studying health information technology in complex adaptive healthcare systems.
        Qual Saf Health Care. 2010; 19: i68-i74
        • Murphy D.R.
        • Meyer A.D.
        • Russo E.
        • et al.
        The burden of inbox notifications in commercial electronic health records.
        JAMA Intern Med. 2016; 176: 559-560
        • Murphy D.R.
        • Reis B.
        • Sittig D.F.
        • et al.
        Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis.
        Am J Med. 2012; 125 (e1–e7): 209
        • Murphy D.R.
        • Reis B.
        • Kadiyala H.
        • et al.
        Electronic health record-based messages to primary care providers: valuable information or just noise?.
        Arch Intern Med. 2012; 172: 283
        • Smith M.
        • Murphy D.
        • Laxmisan A.
        • et al.
        Developing software to “track and catch” missed follow-up of abnormal test results in a complex sociotechnical environment.
        Appl Clin Inform. 2013; 4: 359-375
        • Tarkan S.
        • Plaisant C.
        • Shneiderman B.
        • et al.
        Reducing missed laboratory results: defining temporal responsibility, generating user interfaces for test process tracking, and retrospective analyses to identify problems.
        AMIA Annu Symp Proc. 2011; 2011: 1382-1391
        • Chen E.H.
        • Bodenheimer T.
        Improving population health through team-based panel management: comment on “electronic medical record reminders and panel management to improve primary care of elderly patients.”.
        Arch Intern Med. 2011; 171: 1558-1559
        • Bhise V.
        • Modi V.
        • Kalavar A.
        • et al.
        Patient-reported attributions for missed colonoscopy appointments in two large healthcare systems.
        Dig Dis Sci. 2016; 61: 1853-1861
        • Singh H.
        • Daci K.
        • Petersen L.A.
        • et al.
        Missed opportunities to initiate endoscopic evaluation for colorectal cancer diagnosis.
        Am J Gastroenterol. 2009; 104: 2543-2554
      2. Committee on Diagnostic Error in Health Care, Board on Health Care Services, Institute of Medicine, The National Academies of Sciences, Engineering, and Medicine.
        in: Balogh E.P. Miller B.T. Ball J.R. Improving diagnosis in health care. National Academies Press, Washington, DC2015 (Available from:) (Accessed: August 16, 2016)
        • Resar R.K.
        • Rozich J.D.
        • Classen D.
        Methodology and rationale for the measurement of harm with trigger tools.
        Qual Saf Health Care. 2003; 12: ii39-ii45
        • Classen D.C.
        • Pestotnik S.L.
        • Evans R.S.
        • et al.
        Computerized surveillance of adverse drug events in hospital patients. 1991.
        Qual Saf Health Care. 2005; 14: 221-225
        • Murphy D.R.
        • Laxmisan A.
        • Reis B.A.
        • et al.
        Electronic health record-based triggers to detect potential delays in cancer diagnosis.
        BMJ Qual Saf. 2014; 23: 8-16
        • Murphy D.R.
        • Thomas E.J.
        • Meyer A.N.D.
        • et al.
        Development and validation of electronic health record-based triggers to detect delays in follow-up of abnormal lung imaging findings.
        Radiology. 2015; 277: 81-87
        • Singh H.
        • Giardina T.D.
        • Forjuoh S.N.
        • et al.
        Electronic health record-based surveillance of diagnostic errors in primary care.
        BMJ Qual Saf. 2012; 21: 93-100
        • Murphy D.R.
        • Meyer A.N.D.
        • Bhise V.
        • et al.
        Computerized triggers of big data to detect delays in follow-up of chest imaging results.
        Chest. 2016; 150: 613-620
        • Murphy D.R.
        • Wu L.
        • Thomas E.J.
        • et al.
        Electronic trigger-based intervention to reduce delays in diagnostic evaluation for cancer: a cluster randomized controlled trial.
        J Clin Oncol. 2015; 33: 3560-3567
      3. Evaluation of Colorectal Cancer Screening and Follow-Up in Veterans Health Administration Facilities. VA Office of the Inspector General, Washington, DC2013 (Available from:) (Accessed: June 20, 2017)
        • Howlader N.
        • Noone A.
        • Krapcho M.
        • et al.
        SEER cancer statistics review, 1975-2010.
        National Cancer Institute, Bethesda, MD2013 (Available from:) (Accessed February 20, 2017)
        • Hysong S.J.
        • Sawhney M.K.
        • Wilson L.A.
        • et al.
        Understanding the management of electronic test result notifications in the outpatient setting.
        BMC Med Inform Decis Mak. 2011; 11: 22
        • Danforth K.N.
        • Smith A.E.
        • Loo R.K.
        • et al.
        Electronic clinical surveillance to improve outpatient care: diverse applications within an integrated delivery system.
        EGEMs (Wash DC). 2014; 2: 1056

      Linked Article

      • Correction
        Clinical Gastroenterology and HepatologyVol. 17Issue 6
        • Preview
          Murphy DR, Meyer AND, Vaghani V, Russo E, Sittig DF, Wei L, Wu L, Singh H. Development and validation of trigger algorithms to identify delays in diagnostic evaluation of gastroenterological cancer. Clin Gastroenterol Hepatol 2018;16:90–98.
        • Full-Text
        • PDF