Advertisement

Hepatocellular Carcinoma Prediction Models in Chronic Hepatitis B: A Systematic Review of 14 Models and External Validation

  • Shanshan Wu
    Affiliations
    National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Na Zeng
    Affiliations
    National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Feng Sun
    Affiliations
    Department of Epidemiology and Biostatistics, School of Public Health, Peking University Health Science Centre, Beijing, Mainland China
    Search for articles by this author
  • Jialing Zhou
    Affiliations
    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Xiaoning Wu
    Affiliations
    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Yameng Sun
    Affiliations
    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Bingqiong Wang
    Affiliations
    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Siyan Zhan
    Affiliations
    Department of Epidemiology and Biostatistics, School of Public Health, Peking University Health Science Centre, Beijing, Mainland China
    Search for articles by this author
  • Yuanyuan Kong
    Affiliations
    National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Jidong Jia
    Affiliations
    National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China

    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Hong You
    Correspondence
    Reprint Requests Address requests for reprints to Hong You, PhD, Liver Research Center, Beijing Friendship Hospital, Capital Medical University, 95 Yong-an Road, Xi-Cheng District, Beijing 100050, China. fax: +86-010-63038519.
    Affiliations
    National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China

    Liver Research Center, Beijing Key Laboratory of Translational Medicine in Liver Cirrhosis, Beijing Friendship Hospital, Capital Medical University, Beijing, Mainland China
    Search for articles by this author
  • Hwai-I Yang
    Correspondence
    Hwai-I Yang, PhD, Genomics Research Center, Academia Sinica, 128 Academia Road Section 2, Nankang, Taipei 115, Taiwan. fax: +886-2-27898811.
    Affiliations
    Genomics Research Center, Academia Sinica, Taipei, Taiwan

    Institute of Clinical Medicine, National Yang-Ming University, Taipei, Taiwan

    Graduate Institute of Medicine, College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    Search for articles by this author
Published:March 02, 2021DOI:https://doi.org/10.1016/j.cgh.2021.02.040

      Background & Aims

      The aim of our study was to characterize the performance of hepatocellular carcinoma (HCC) prediction models in chronic hepatitis B (CHB) patients through meta-analysis followed by external validation.

      Methods

      We performed a systematic review and meta-analysis of current literature, followed by external validation in independent multi-center cohort with 986 patients with CHB undergoing entecavir treatment (median follow-up: 4.7 years). Model performance to predict HCC within 3, 5, 7, and 10 years was assessed using area under receiver operating characteristic curve (AUROC) and calibration index. Subgroup analysis were conducted by treatment status, cirrhotic, race and baseline alanine aminotransferase.

      Results

      We identified 14 models with 123,885 patients (5,452 HCC cases), with REACH-B, CU-HCC, GAG-HCC, PAGE-B and mPAGE-B models being broadly externally validated. Discrimination was generally acceptable for all models, with pooled AUC ranging from 0.70 (95% CI, 0.63-0.76 for REACH-B) to 0.83 (95% CI, 0.78-0.87 for REAL-B) for 3-year, 0.68 (95% CI, 0.64-0.73 for REACH-B) to 0.81 (95% CI, 0.77-0.85 for REAL-B) for 5-year and 0.70 (95% CI, 0.58-0.80 for PAGE-B) to 0.81 (95% CI, 0.78-0.84 for REAL-B and 0.77-0.86 for AASL-HCC) for 10-year prediction. However, calibration performance was poorly reported in most studies. In external validation cohort, REAL-B showed highest discrimination with 0.76 (95% CI, 0.69-0.83) and 0.75 (95% CI, 0.70-0.81) for 3 and 5-year prediction. The REAL-B model was also well calibrated in the external validation cohort (3-year Brier score 0.066). Results were consistent in subgroup analyses.

      Conclusions

      In a systematic review of available HCC models, the REAL-B model exhibited best discrimination and calibration.

      Keywords

      Abbreviations used in this paper:

      AFP (α-fetoprotein), ALB (albumin), ALT (alanine aminotransferase), AUROC (area under the receiver-operating characteristic curve), BFH (Beijing Friendship Hospital), BS (Brier score), CHB (chronic hepatitis B), CI (confidence interval), E (expected), HBeAg (hepatitis B e antigen), HBV (hepatitis B virus), HCC (hepatocellular carcinoma), LSM (liver stiffness measurement), O (observed), PLT (platelet)

       Background

      • Hepatocellular carcinoma (HCC) screening in chronic hepatitis B (CHB) patients is burdensome and costly.
      • Accurate risk prediction model could help improve efficiency and save medical resources, however, there is no systematic review to summarize and evaluate the comparing external performance across all different models.

       Findings

      • A total of 14 HCC risk prediction models in CHB patients exist, mainly built in untreated patients, with only minority being externally validated and few reporting calibration performance in most studies.
      • All 14 models showed accepted discrimination, with REAL-B model exhibiting best discrimination and calibration, either in cirrhotic or elevated baseline ALT level.

       Implications for patient care

      • REAL-B model may have potential clinical utility for identifying high-risk CHB patients in HCC surveillance.
      Chronic hepatitis B (CHB) virus infection remains the leading cause of hepatocellular carcinoma (HCC) globally.
      • Singal A.G.
      • El-Serag H.B.
      Hepatocellular carcinoma from epidemiology to prevention: translating knowledge into practice.
      • World Health Organization
      Guidelines for the Prevention, Care and Treatment of Persons With Chronic Hepatitis B Infection.
      Although nucleos(t)ide analogs have been proved to suppress hepatitis B virus (HBV) replication and decrease HCC risk, they do not completely eliminate HCC occurrence.
      • World Health Organization
      Guidelines for the Prevention, Care and Treatment of Persons With Chronic Hepatitis B Infection.
      ,
      • Cho J.Y.
      • Paik Y.H.
      • Sohn W.
      • et al.
      Patients with chronic hepatitis B treated with oral antiviral therapy retain a higher risk for HCC compared with patients with inactive stage disease.
      Currently, most CHB management guidelines
      • Terrault N.A.
      • Lok A.S.
      • McMahon B.J.
      • et al.
      Update on prevention, diagnosis, and treatment of chronic hepatitis B: AASLD 2018 hepatitis B guidance.
      • Sarin S.K.
      • Kumar M.
      • Lau G.K.
      • et al.
      Asian-Pacific clinical practice guidelines on the management of hepatitis B: a 2015 update.
      • Lampertico P.
      • Agarwal K.
      • Berg T.
      • et al.
      EASL 2017 Clinical Practice Guidelines on the management of hepatitis B virus infection.
      recommend HCC surveillance every 6–12 months, it is still critically important to accurately identify patients with high risk of HCC occurrence.
      Risk prediction models might estimate individualized probability of developing HCC, which could help physicians improve efficiency and implementation of screening surveillance strategy.
      • Wong V.W.
      • Janssen H.L.
      Can we use HCC risk scores to individualize surveillance in chronic hepatitis B infection?.
      Several models
      • Yuen M.F.
      • Tanaka Y.
      • Fong D.Y.
      • et al.
      Independent risk factors and predictive score for the development of hepatocellular carcinoma in chronic hepatitis B.
      • Yang H.I.
      • Sherman Su J.
      • et al.
      Nomograms for risk of hepatocellular carcinoma in patients with chronic hepatitis B virus infection.
      • Wong V.W.
      • Chan S.L.
      • MO F.
      • et al.
      Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
      • Yang H.I.
      • Yuen M.F.
      • Chan H.L.
      • et al.
      Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
      • Wong G.L.
      • Chan H.L.
      • Wong C.K.
      • et al.
      Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B.
      • Lee H.W.
      • Yoo E.J.
      • Kim B.K.
      • et al.
      Prediction of development of liver-related events by transient elastography in hepatitis B patients with complete virological response on antiviral therapy.
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      • Poh Z.
      • Shen L.
      • Yang H.I.
      • et al.
      Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B.
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      exist for predicting HCC occurrence in CHB patients, including earlier models built in untreated patients (GAG-HCC [guide with age, gender, HBV DNA, core promoter mutations and cirrhosis-hepatocellular carcinoma], NGM-HCC [nomogram-hepatocellular carcinoma], and REACH-B [risk estimation for hepatocellular carcinoma in chronic hepatitis B]), recent models built in treated patients (modified REACH-B [mREACH-B], PAGE-B [platelet, age, gender and HBV], modified PAGE-B [mPAGE-B], CAMD [cirrhosis, age, male sex, and diabetes mellitus], and REAL-B [real-world effectiveness from the Asia-Pacific Rim for HBV risk score]), and other models based on mixed patients with different treatment proportion (CU-HCC [Chinese University-hepatocellular carcinoma], LSM-HCC [liver stiffness measurement–hepatocellular carcinoma], and RWS-HCC [real-world risk score-hepatocellular carcinoma]). However, none of them was recommended by guidelines to be widely used in clinical practice. Some crucial case mix across these models, such as cirrhosis proportion and baseline alanine aminotransferase (ALT) level, appeared significantly different.
      It was also noticed only a few of those models have been externally validated in multiple cohorts,
      • Yuen M.F.
      • Tanaka Y.
      • Fong D.Y.
      • et al.
      Independent risk factors and predictive score for the development of hepatocellular carcinoma in chronic hepatitis B.
      ,
      • Wong V.W.
      • Chan S.L.
      • MO F.
      • et al.
      Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
      • Yang H.I.
      • Yuen M.F.
      • Chan H.L.
      • et al.
      Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
      • Wong G.L.
      • Chan H.L.
      • Wong C.K.
      • et al.
      Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B.
      and some recently published models have not been externally validated yet.
      • Poh Z.
      • Shen L.
      • Yang H.I.
      • et al.
      Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B.
      ,
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      Even within those validation studies, limited sample sizes and HCC events could lead to conflicting evidence and relatively low statistical power.
      • Coffin C.S.
      • Rezaeeaval M.
      • Pang J.X.
      • et al.
      The incidence of hepatocellular carcinoma is reduced in patients with chronic hepatitis B on long-term nucleos(t)ide analogue therapy.
      • Arends P.
      • Sonneveld M.J.
      • Zoutendijk R.
      • et al.
      Entecavir treatment does not eliminate the risk of hepatocellular carcinoma in chronic hepatitis B: limited role for risk scores in Caucasians.
      • Kim W.R.
      • Loomba R.
      • Berg T.
      • et al.
      Impact of long-term tenofovir disoproxil fumarate on incidence of hepatocellular carcinoma in patients with chronic hepatitis B.
      • Ahn J.
      • Lim J.K.
      • Lee H.M.
      • et al.
      Lower observed hepatocellular carcinoma incidence in chronic hepatitis B patients treated with entecavir: results of the ENUMERATE study.
      Thus, it is necessary to synthesize evidence from all external validation studies for the same model to assess that model’s performance across different populations with relatively large sample size. Furthermore, so far there is also lack of evidence comparing the performance across all different models with head-to-head validation in the same cohort.
      In this study, we aimed to systematically identify all published HCC prediction models for CHB patients and externally validate the performance of these models through a meta-analysis followed by an independent external validation of all models using our multicenter cohort with representative Chinese CHB patients undergoing antiviral therapy.

      Materials and Methods

      The systematic review and meta-analysis was registered on PROSPERO (CRD42020171960) and conducted according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist (Supplementary Appendix 1).

       Systematic Search and Identification of Published Models

       Data Sources and Searches

      Medline, Embase, and the Cochrane Library were searched until October 7, 2019, with keywords hepatocellular carcinoma, hepatitis B, prognostic, and prediction model, etc. (Supplementary Appendix 2). Additionally, we newly checked reference list for all relevant papers to identify any additional studies.

       Study Selection

      Studies presenting a formal prediction model to provide individualized HCC prediction in CHB patients were included in our meta-analysis. Estimates of 3-, 5-, 7-, and 10-year HCC risk was focused. Both model development and external validation studies were retrieved. Study eligibility was assessed independently by S.W. and N.Z. in duplicate. Any discrepancies were resolved by consensus.

       Data Extraction and Quality Assessment

      Data were extracted following guidance of the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist.
      • Moons K.G.M.
      • de Groot J.A.H.
      • Bouwmeester W.
      • et al.
      Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.
      For model validation studies, data were extracted, including study information (author, publication year, region, sample size, HCC cases, follow-up duration, validated HCC models), population characteristics, medical history, and laboratory variables (race, mean/median age, proportion of cirrhosis, antiviral therapy and hepatitis B e antigen [HBeAg] positivity, baseline HBV DNA, platelet [PLT], albumin [ALB], ALT, total bilirubin, α-fetoprotein [AFP], liver stiffness measurement [LSM], mean score with SD and model performance [area under the receiver-operating characteristic curve (AUROC) with 95% confidence interval (CI) or standard error for discrimination], total number of observed [O] or expected [E] events for calibration).
      For model development studies, additional information to perform external validation was extracted: standard deviation of age, definition of each predictors, parameter estimation, risk score formula, and model performance. For models constructed with Cox proportional hazards regression and logistic regression, disease-free probability at a specific time point and intercept were extracted.
      S.W., N.Z., and F.S. extracted data independently in duplicate. For studies without available model performance data, we emailed twice for data request. Risk of bias was critically appraised according to PROBAST (Prediction model Risk Of Bias ASsessment Tool),
      • Wolff R.F.
      • Moons K.G.M.
      • Riley R.D.
      • et al.
      PROBAST: a tool to assess the risk of bias and applicability of prediction model studies.
      which characterizes the quality on basis of study participants, predictors, outcome, and statistical analysis.

       Validation Cohort

      An independent, multicenter, prospective cohort conducted in 22 hospitals in China was used to validate eligible models (Beijing Friendship Hospital [BFH] cohort). This cohort comprised 986 treatment-naïve CHB patients 18–65 years of age enrolled between 2013 and 2015.
      • Wu S.
      • Kong Y.
      • Piao H.
      • et al.
      On-treatment changes of liver stiffness at week 26 could predict 2-year clinical outcomes in HBV-related compensated cirrhosis.
      All patients receiving entecavir 0.5 mg/d–based treatment were assessed for demographic characteristics (age, sex, family history of HCC, diabetes), alcohol intake, blood cell count, liver function test, HBV DNA, HBeAg, AFP, liver ultrasonography, and LSM at baseline and every 26 weeks. Liver biopsies were performed in eligible patients at baseline to evaluate fibrosis stage. We set the end of follow-up for HCC occurrence as of January 1, 2020. The median follow-up period was 4.7 years.
      Therefore, all predictors of eligible models were available in the BFH cohort. Cirrhosis at baseline was diagnosed by (1) liver biopsy with Metavir fibrosis score equal to 4; (2) endoscopy with esophageal varices, excluding noncirrhotic portal hypertension; and (3) when biopsy and endoscopy could not be performed, it should meet 2 of following 4 criteria: (a) ultrasonography, computed tomography or magnetic resonance indicated imaging changes in liver morphology, such as nodules in hepatic parenchyma and serrated change on liver surface; (b) PLT count <100 × 109/L; (c) ALB <35 g/L, or international normalized ratio >1.3; or (d) LSM >12.4 kPa when ALT was <5 times the upper limit of normal. Thus, diagnostic criteria for cirrhosis in the BFH cohort was similar to other studies.
      Regarding HCC diagnosis, dynamic imaging including contrast-enhanced computed tomography or magnetic resonance was performed for characterization at detection of a newly developed hepatic nodule. Diagnosis and staging of HCC were based on recommendations of American Association for the Study of Liver Diseases.
      • Bruix J.
      • Sherman M.
      American Association for the Study of Liver Diseases
      Management of hepatocellular carcinoma: an update.

       Data Synthesis and Analysis

       Methods for Meta-Analysis of Each Prediction Model

      Only model external validation studies were included in meta-analysis. AUC and total O:E ratio with 95% CI were calculated as effect measures for model discrimination and calibration. The 95% prediction interval was calculated for heterogeneity. To account for uncertainty in heterogeneity, a random-effects model was implemented using restricted maximum likelihood estimation and Hartung-Knapp-Sidik-Jonkman method when calculating 95% CIs for average performance.
      • Debray T.P.
      • Damen J.A.
      • Riley R.D.
      • et al.
      A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes.
      Additionally, to investigate potential sources of heterogeneity for model’s performance, subgroup analysis was conducted by cirrhotic status (cirrhosis or noncirrhosis), antiviral therapy or not (treated or untreated), race (Asian or non-Asian), and baseline ALT level (high or normal). We emailed twice for unreported subgroup data. Besides, sensitivity analysis was conducted to validate results robustness by omitting with high-risk validation studies.

      Methods for Independent External Validation

      Descriptive statistics were used to detail BFH cohort baseline characteristics. Mean ± SD or median (range) was used for continuous variables and percentages were used for categorical variables. The Kaplan-Meier method was used to calculate cumulative incidence of HCC. Based on the current follow-up in the BFH cohort, the prediction time point for all models was restricted to 3 and 5 years.
      Model discrimination was assessed in the BFH cohort using time-dependent AUROC with 95% CI through inverse probability of censoring weighting, followed by head-to-head comparison among all models. Calibration was evaluated graphically by plotting mean observed vs predicted probability in deciles of predicted probability. Meanwhile, the Brier score (BS) was also calculated to assess models’ calibration. Only 6 models (REACH-B, REAL-B, PAGE-B, mPAGE-B, CAMD, and AASL-HCC [age, sex, albumin, and liver cirrhosis for hepatocellular carcinoma]) were available to assess calibration finally. To further assess whether model’s performance was impacted by patients’ characteristics, similar subgroup analysis was conducted in the BFH cohort by cirrhotic status and baseline ALT level.
      For missing predictors, complete-case analysis was conducted. Overall, 14.9%, 2.9%, 2.5%, 2.0%, 2.0%, 1.3%, and 1.1% were missing in baseline LSM, AFP, HBV DNA, ALB, PLT, total bilirubin, and ALT, respectively. Sensitivity analysis was performed to validate model performance after multiple imputation with all 7 variables included, 5 imputed datasets created, and Rubin rule estimation.
      For prediction models reporting cutoffs, we validated these cutoffs in the BFH cohort by calculating common diagnostic accuracy measures. Besides, efficiency and failure rate were calculated from clinical practice point of view. Efficiency was defined as proportion of patients in whole cohort stratified to group with low HCC predicted probability. Failure rate was defined as proportion of patients with low predicted probability ultimately developing HCC later, calculating through Kaplan-Meier method. Difference of failure rate with 95% CI was calculated by bootstrap of 1000 samples.
      All analyses were conducted using SAS software version 9.4 (SAS Institute, Cary, NC) and R version 3.6.2 (metamisc, timeROC package, ipred, ggplot2 and forestplot package) (R Foundation for Statistical Computing, Vienna, Austria).

      Results

       Identification and Characteristics of Included Studies for Model Development and Validation

      After screening 1135 publications, 30 publications
      • Yuen M.F.
      • Tanaka Y.
      • Fong D.Y.
      • et al.
      Independent risk factors and predictive score for the development of hepatocellular carcinoma in chronic hepatitis B.
      • Yang H.I.
      • Sherman Su J.
      • et al.
      Nomograms for risk of hepatocellular carcinoma in patients with chronic hepatitis B virus infection.
      • Wong V.W.
      • Chan S.L.
      • MO F.
      • et al.
      Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
      • Yang H.I.
      • Yuen M.F.
      • Chan H.L.
      • et al.
      Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
      • Wong G.L.
      • Chan H.L.
      • Wong C.K.
      • et al.
      Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B.
      • Lee H.W.
      • Yoo E.J.
      • Kim B.K.
      • et al.
      Prediction of development of liver-related events by transient elastography in hepatitis B patients with complete virological response on antiviral therapy.
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      • Poh Z.
      • Shen L.
      • Yang H.I.
      • et al.
      Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B.
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      • Coffin C.S.
      • Rezaeeaval M.
      • Pang J.X.
      • et al.
      The incidence of hepatocellular carcinoma is reduced in patients with chronic hepatitis B on long-term nucleos(t)ide analogue therapy.
      • Arends P.
      • Sonneveld M.J.
      • Zoutendijk R.
      • et al.
      Entecavir treatment does not eliminate the risk of hepatocellular carcinoma in chronic hepatitis B: limited role for risk scores in Caucasians.
      • Kim W.R.
      • Loomba R.
      • Berg T.
      • et al.
      Impact of long-term tenofovir disoproxil fumarate on incidence of hepatocellular carcinoma in patients with chronic hepatitis B.
      • Ahn J.
      • Lim J.K.
      • Lee H.M.
      • et al.
      Lower observed hepatocellular carcinoma incidence in chronic hepatitis B patients treated with entecavir: results of the ENUMERATE study.
      ,
      • Wong G.L.
      • Chan H.L.
      • Chan H.Y.
      • et al.
      Accuracy of risk scores for patients with chronic hepatitis B receiving entecavir treatment.
      • Chen W.
      • Chen J.
      • Zheng Q.
      • et al.
      Validation study of prediction models of hepatitis B virus-related hepatocellular carcinoma.
      • Kim G.A.
      • Lee H.C.
      • Kim M.J.
      • et al.
      Incidence of hepatocellular carcinoma after HBsAg seroclearance in chronic hepatitis B patients: a need for surveillance.
      • Abu-Amara M.
      • Cerocchi O.
      • Malhi G.
      • et al.
      The applicability of hepatocellular carcinoma risk prediction scores in a North American patient population with chronic hepatitis B infection.
      • Tawada A.
      • Chiba T.
      • Saito T.
      • et al.
      Utility of prediction scores for hepatocellular carcinoma in patients with chronic hepatitis B treated with nucleos(t)ide analogues.
      • Papatheodoridis G.V.
      • Dalekos G.N.
      • Yurdaydin C.
      • et al.
      Incidence and predictors of hepatocellular carcinoma in Caucasian chronic hepatitis B patients receiving entecavir or tenofovir.
      • Kim M.N.
      • Hwang S.G.
      • Rim K.S.
      • et al.
      Validation of PAGE-B model in Asian chronic hepatitis B patients receiving entecavir or tenofovir.
      • Yang H.I.
      • Tseng T.C.
      • Liu J.
      • et al.
      Incorporating serum level of hepatitis b surface antigen or omitting level of hepatitis B virus DNA does not affect calculation of risk for hepatocellular carcinoma in patients without cirrhosis.
      • Brouwer W.P.
      • van der Meer A.J.P.
      • Boonstra A.
      • et al.
      Prediction of long-term clinical outcome in a diverse chronic hepatitis B population: Role of the PAGE-B score.
      • Daheim M.
      • Lang S.
      • Goeser T.
      • et al.
      Real-world risk score for hepatocellular carcinoma risk prediction in CHBV: a validation outside of Asia.
      • Riveiro-Barciela M.
      • Tabernero D.
      • Calleja J.L.
      • et al.
      Effectiveness and safety of entecavir or tenofovir in a Spanish cohort of chronic hepatitis B Patients: validation of the Page-B score to predict hepatocellular carcinoma.
      • Seo Y.S.
      • Jang B.K.
      • Um S.H.
      • et al.
      Validation of risk prediction models for the development of HBV-related HCC: a retrospective multi-center 10-year follow-up cohort study.
      • Jeon M.Y.
      • Lee H.W.
      • Kim S.U.
      • et al.
      Feasibility of dynamic risk prediction for hepatocellular carcinoma development in patients with chronic hepatitis B.
      • Yip T.C.
      • Wong G.L.
      • Wong V.W.
      • et al.
      Reassessing the accuracy of PAGE-B-related scores to predict hepatocellular carcinoma development in patients with chronic hepatitis B.
      with 123,885 CHB patients and 5452 HCC cases met the inclusion criteria (Supplementary Appendix 3). As shown in Supplementary Appendix 4, 12 model development studies containing 14 eligible models with totally 43,095 patients and 1687 HCC cases were identified. Twenty-nine model validation studies with 87,832 patients and 4196 HCC cases validated 13 models except for the mREACH-BI. Of these, the REACH-B, CU-HCC, GAG-HCC, PAGE-B, and mPAGE-B were frequently externally validated in 24,344 patients (1582 HCC cases), 21,830 patients (1220 HCC cases), 17,814 patients (952 HCC cases), 63,556 patients (2839 HCC cases), and 45,732 patients (2555 HCC cases), respectively. Other models that were externally validated included the NGM1-HCC, NGM2-HCC, LSM-HCC, mREACH-BII, AASL-HCC, RWS-HCC, CAMD, and REAL-B.
      Of all, 13 models were developed in Asian populations, except for the PAGE-B, which was developed in a Caucasian population. The median sample size and HCC cases for the derivation cohort was 1180 (range, 192–23,851) and 80 (range, 15–596), respectively. The majority were developed for predicting 3-, 5-, and 10-year HCC occurrence. The number of predictors ranged from 3 to 7 (Table 1). Not surprisingly, age was the most common predictor in all models, followed by sex (12 models), cirrhosis (6 models), HBeAg (5 models), ALB (4 models), PLT (3 models), and LSM (3 models). Specific predictors and categories, derived scores, and risk formulas of each model ware shown in Supplementary Appendix 5. The populations for deriving different models were heterogeneous, particularly in antiviral treatment and cirrhosis status (Table 2). The GAG-HCC, NGM1-HCC, NGM2-HCC and REACH-B were based on untreated patients, while the mREACH-BI, mREACH-BII, PAGE-B, mPAGE-B, AASL-HCC, CAMD, and REAL-B were based on treated patients. The CU-HCC, LSM-HCC, and RWS-HCC contained mixed patients with a treatment proportion ranging from 15% to 36%. Regarding cirrhosis status, REACH-B was the only model based on noncirrhotic population, while other 13 models were developed in populations with cirrhosis proportion ranging from 15% to 47%.
      Table 1Predictor Variables Contained Within 14 Identified HCC Risk Prediction Models
      ModelDemographicMedical historyLifestyleLaboratory VariablesModel Score Cutoff Point
      AgeSexCirrhosisHCC Family historyAlcoholHBeAgHBV DNAALTPLTALBTBILLSMAFP
      REACH-B
      mREACH-BI
      mREACH-BII
      GAG-HCCLow: 0–100; High: >100
      CU-HCCLow: 0–4; Intermediate: 5–19; High: ≥19
      LSM-HCCLow: 0–10;

      High: 11–30
      PAGE-BLow: 0–9; Intermediate: 10–17; High: ≥18
      mPAGE-BLow: 0–8; Intermediate: 9–12; High: ≥13
      RWS-HCCLow: <4.5; High: ≥4.5
      AASL-HCCLow: 0–5; Intermediate: 6–19; High: ≥20
      NGM1-HCC
      NGM2-HCC
      CAMDLow: 0–7; Intermediate: 8–13; High: >13
      REAL-BLow: 0–3; Intermediate: 4–7; High: 8–13
      NOTE. The star indicates a continuous variable and a circle indicates a categorical variable.
      AASL-HCC, age, sex, albumin, and liver cirrhosis for hepatocellular carcinoma; AFP, α-fetoprotein; ALB, albumin; ALT, alanine aminotransferase; CAMD, cirrhosis, age, male sex, and diabetes mellitus; CU-HCC, Chinese University-hepatocellular carcinoma; DM, diabetes mellitus; DNA, deoxyribonucleic acid; GAG-HCC, guide with age, gender, HBV DNA, core promoter mutations and cirrhosis-hepatocellular carcinoma; HBeAg, hepatitis B e antigen; HBV, hepatitis B virus infection; HCC, hepatocellular carcinoma; LSM-HCC, liver stiffness measurement-hepatocellular carcinoma; NGM-HCC, nomogram-hepatocellular carcinoma; PAGE-B, platelet, age, gender and HBV; mPAGE-B, modified platelet, age, gender and HBV; PLT, platelet; REACH-B, risk estimation for hepatocellular carcinoma in chronic hepatitis B; REAL-B, real-world effectiveness from the Asia-Pacific Rim for HBV risk score; RWS-HCC, real-world risk score for hepatocellular carcinoma; TBIL, total bilirubin.
      Table 2Baseline Characteristics of BFH Validation Cohort and Each Model Original Derivation Cohort
      VariablesBFH Validation Cohort (n = 986)REACH-B (n = 3584)12mREACH-B (n = 192)14GAG-HCC (n = 820)9CU-HCC (n = 1005)11LSM-HCC (n = 1035)13PAGE-B (n = 1325)15mPAGE-B (n = 2001)17RWS-HCC (n = 538)16AASL-HCC (n = 944)19NGM-HCC (n = 2435)10CAMD (n = 23,851)18REAL-B (n = 5365)20
      Demographic characteristics
      Age, y43.7 ± 11.145.7 ± 9.849.0 ± 10.440.6 ± 17.448.0 ± 7.046.0 ± 12.052.0 ± 15.650 ± 11.156.4 ± 12.150 ± 11.945.8 ± 10.347.5 ± 13.948.4 ± 12.7
      Male735 (74.5)2198 (61.3)134 (69.8)573 (69.9)681 (67.8)661 (63.9)923 (70)1282 (64.1)337 (62.6)586 (62.1)1524 (62.6)17649 (74.0)3710 (69.2)
      Asian987 (100)3584 (100)192 (100)820 (100)1005 (100)1035 (100)0 (0)2001 (100)538 (100)944 (100)2435 (100)23851 (100)5356 (100
      Medical history
      DM35 (3.5)NR12 (6.3)NRNRNRNRNRNRNRNR2950 (12.37)544 (10.9)
      Family history of HCC128 (13.0)NRNRNRNRNRNRNRNRNR107 (4.4)NRNR
      Cirrhosis663 (67.2)0 (0.0)90 (46.9)124 (15.1%)383 (38.1)331 (32.0)269 (20)383 (19.1)80 (14.9)371 (39.3)46 (1.9)6308 (26.5)1085 (20.2)
      Antiviral treatment987 (100)0 (0.0)192 (100)0 (0)152 (15.1)390 (37.7)1325 (100)2001 (100)97 (18)944 (100)0 (0)23851 (100)5356 (100)
      Lifestyle factors
      Alcohol intake226 (22.9)NR50 (26.0)NRNRNRNRNRNRNR302 (12.4)NR967 (19.4)
      Laboratory variables
      HBeAg positive479 (48.5)3039 (84.8)100 (52.1)356 (43.4)NR256 (24.7)210 (16)678 (33.9)167 (31)528 (55.9)366 (15)NR1886 (37.4)
      HBV DNA, log IU/mL5.4 ± 1.9NR0 (0)5.2 ± 2.63.9 ± 1.04.9 ± 2.15.6 ± 2.03.0 ± 3.9NR5.5 ± 1.9NRNR5.2 ± 2.6
      ALT, IU/L58.9 (37.8–108.0566.5 ± 98.426.0 (18.0–36.0)47 (5–1251)NR69.0 ± 118.0NR57 (30–136)NR96 (53–194)NRNR69 (38–171)
      Normal ALT343 (34.8)3388 (94.5)NR409 (49.9)143 (14.2)682 (66)518 (42)NRNRNR2288 (94.0)NRNR
      PLT, 109/L116.0 (76.0–164.0∖NRNR175 (55–290)NR210.0 ± 62.0191 (76)158 (115–202)NR158 (115–202)NRNR172.8 ± 71.3
      ALB, g/dL42.0 (38.4–45.3)NR44.0 (42.0–47.0)44 (22–56)39.0 ± 4.044.0 ± 3.0NR42 (39–44)NR40 (35–43)NRNR41 ± 5.6
      TBIL, μmol/L16.8 (12.4–23.7)NR15.4 (12.0–18.8)11 (1–77)10.0 ± 9.016.0 ± 17.0NR12.0 (17.1–22.2)NR15.4 (12.0–22.2)NRNR12.0 (8.6–17.1)
      LSM, kPa15.9 (10.7–23.8)NR8.8 (5.9–13.6)NRNR8.4 ± 6.1NRNRNRNRNRNRNR
      AFP, μg/L5.7 (2.9–15.3)NR3.0 (2.0–4.9)NRNR8.0 ± 38.0NRNRNR5.5 (3–14.8)NRNR4.6 (3.0–10.0)
      Risk prediction model scores
      REACH-B10.45 ± 2.79NRNRNRNRNRNR9 ± 3.7NRNRNRNRNR
      mREACH-BI8.12 ± 2.27NRNRNRNRNRNRNRNRNRNRNRNR
      mREACH-BII9.62 ± 2.71NRNRNRNRNRNRNRNRNRNRNRNR
      GAG-HCC90.32 ± 22NRNRNRNRNRNR74.0 ± 19.3NRNRNRNRNR
      CU-HCC16.01 ± 10.8NRNRNRNRNRNR4.5 (2.5–17.3)NRNRNRNRNR
      LSM-HCC15.6 ± 10.89NRNRNRNRNRNRNRNRNRNRNRNR
      PAGE-B14.77 ± 4.63NRNRNRNRNRNR14.0 ± 5.9NRNRNRNRNR
      mPAGE-B10.55 ± 3.34NRNRNRNRNRNRNRNRNRNRNRNR
      RWS-HCC4.69 ± 2.04NRNRNRNRNRNRNRNRNRNRNRNR
      AASL-HCC13.9 ± 6.64NRNRNRNRNRNRNRNRNRNRNRNR
      NGM1-HCC8.24 ± 2.51NRNRNRNRNRNRNRNRNRNRNRNR
      NGM2-HCC11.2 ± 2.68NRNRNRNRNRNRNRNRNRNRNRNR
      CAMD10.55 ± 5.16NRNRNRNRNRNRNRNRNRNRNRNR
      REAL-B5.28 ± 2.17NRNRNRNRNRNRNRNRNRNRNRNR
      HCC occurrence
      Median follow-up period, y4.7123.65.6105.84.244.94.3112.25.5
      HCC events56131154010538511324256103596378
      NOTE. Values are mean ± SD, n (%), or median (range).
      AASL-HCC, age, sex, albumin, and liver cirrhosis for hepatocellular carcinoma; AFP, α-fetoprotein; ALB, albumin; ALT, alanine aminotransferase; BFH, Beijing Friendship Hospital; CAMD, cirrhosis, age, male sex, and diabetes mellitus; CU-HCC, Chinese University-hepatocellular carcinoma; DM, diabetes mellitus; DNA, deoxyribonucleic acid; GAG-HCC, guide with age, gender, HBV DNA, core promoter mutations and cirrhosis-hepatocellular carcinoma; HBeAg, hepatitis B e antigen; HBV, hepatitis B virus infection; HCC, hepatocellular carcinoma; LSM, liver stiffness measurement; LSM-HCC, liver stiffness measurement-hepatocellular carcinoma; mPAGE-B, modified platelet, age, gender and HBV; mREACH-B, modified risk estimation for hepatocellular carcinoma in chronic hepatitis B; NGM-HCC, nomogram-hepatocellular carcinoma; NR, not reported; PAGE-B, platelet, age, gender and HBV; PLT, platelet; REACH-B, risk estimation for hepatocellular carcinoma in chronic hepatitis B; REAL-B, real-world effectiveness from the Asia-Pacific Rim for HBV risk score; RWS-HCC, real-world risk score for hepatocellular carcinoma; TBIL, total bilirubin.
      Regarding study quality, data sources and inclusion and exclusion criteria were appropriate in most studies (96.7% and 100%, respectively). The definition, timing, and measurement predictors and outcome were generally at low risk of bias. However, 12 (40%) of studies were at high risk in analysis, mainly owing to small HCC numbers, inconsistency between predictors with assigned weights in final model and reported multivariable analysis, and inappropriate assessment of model performance. A total of 22 (73.3%) of studies did not evaluate calibration performance. As for other analysis signaling questions, such as continuous or categorical predictors handling, missing data management, and data complexity consideration, most studies performed well, except for RWS-HCC based on binary logistic regression not accounting for censoring data. Overall, risk of bias was relatively low.

       Characteristics of the BFH External Validation Cohort

      In the BFH validation cohort, HCC was diagnosed in 56 of 986 patients during the follow-up period. The 5-year cumulative incidence of HCC was 7.5%. Table 2 shows the baseline characteristics of the BFH cohort as well as 12 derivation cohorts in all model development studies, which was generally comparable. However, there was variability across proportion of cirrhosis and antiviral treatment, HBeAg positive, biochemical indexes, and LSM. Overall, BFH cohort comprised treatment-naïve patients undergoing antiviral therapy with the largest proportion of cirrhosis (67.2%), which was representative in the era of antiviral treatment.

       Results of Meta-Analysis of Model Validation Studies

      As shown in Figure 1, Table 3, and Supplementary Appendix 7, discrimination was generally acceptable for all models, with a summary AUROC ranging from 0.70 (for REACH-B: 95% CI, 0.63–0.76) to 0.85 (REAL-B: 95% CI, 0.78–0.87; for mREACH-BII: 95% CI, 0.79–0.85) for 3-year prediction, 0.68 (REACH-B: 95% CI, 0.64–0.73) to 0.81 (REAL-B: 95% CI, 0.77–0.85) for 5-year prediction, and 0.70 (PAGE-B: 95% CI, 0.58–0.80) to 0.81 (REAL-B: 95% CI, 0.78–0.84; AASL-HCC: 95% CI, 0.77–0.86) for 10-year prediction. CAMD also demonstrated good discrimination with the narrowest CI (for 3-year prediction: 0.75; 95% CI, 0.72–0.77; for 5-year prediction: 0.76; 95% CI, 0.74–0.77). PAGE-B did not show excellent discrimination performance than other models with AUROC equal to 0.74 (95% CI, 0.70–0.77), 0.73 (95% CI, 0.69–0.77), and 0.70 (95% CI, 0.58–0.80) for 3-, 5-, and 10-year prediction. No significant difference in discrimination was detected across different prediction time-points for the same model. However, calibration performance was poorly reported in most development and external validation studies, resulting in barely informative summary of O:E ratio for most models because of wide 95% prediction intervals. The REACH-B, CU-HCC, PAGE-B, and mPAGE-B were validated broadly, while the CAMD and REAL-B were externally validated in only 1 cohort with good calibration. Particularly, the mPAGE-B seemed overestimate 3- and 5-year HCC risk with summary total O:E ratio of 0.26 (95% CI, 0.20–0.34) for 3-year prediction and 0.306 (95% CI, 0.30–0.31) for 5-year prediction. Additionally, results of sensitivity analyses were similar (Supplementary Appendix 8).
      Figure thumbnail gr1
      Figure 1Discrimination and calibration results of meta-analysis of HCC risk prediction models at 3, 5, 7, and 10 years. (A) Discrimination and (B) calibration. Discrimination was generally acceptable for all models, with the REAL-B and CAMD achieving best performance with narrowest CI. In contrast, calibration performance was poorly reported in most studies, resulting in less informative summary O:E ratio for most models. The CAMD and REAL-B were externally validated in only 1 cohort with good calibration performance, while the mPAGE-B seemed to overestimate 3- and 5-year HCC risk.
      Table 3HCC Risk Model Discrimination in Original Derivation Cohort, Meta-Analysis of External Validation Studies, and BFH Validation Cohort
      Original Derivation CohortMeta-Analysis of External Validation StudiesBFH Validation Cohort
      Model3-y AUROC (95% CI)5-y AUROC (95% CI3-y AUROC (95% CI5-y AUROC (95% CI3-y AUROC (95% CI5-y AUROC (95% CI
      REACH-B0.86 (0.84–0.88)0.86 (0.85–0.88)0.70 (0.63–0.76)0.68 (0.64–0.73)0.68 (0.51–0.85)0.60 (0.51–0.69)
      mREACH-BI0.81 (0.68–0.93)NRNRNR0.72 (0.61–0.83)0.63 (0.53–0.72)
      mREACH-BII0.81 (0.71–0.91)NR0.83 (0.79–0.85)0.79 (0.69–0.86)0.73 (0.64–0.82)0.64 (0.57–0.72)
      GAG-HCCNR0.87 (0.82–0.93)0.74 (0.64–0.82)0.76 (0.71–0.81)0.74 (0.62–0.87)0.74 (0.65–0.83)
      CU-HCCNRNR0.74 (0.67–0.80)0.73 (0.69–0.77)0.65 (0.56–0.74)0.65 (0.59–0.71)
      LSM-HCC0.83 (0.76–0.91)0.83 (0.77–0.90)0.80 (0.63–0.90)0.74 (0.68–0.79)0.74 (0.62–0.85)0.67 (0.58–0.76)
      PAGE-BNR0.820.74 (0.70–0.77)0.73 (0.69–0.77)0.72 (0.56–0.89)0.63 (0.52–0.75)
      mPAGE-BNR0.82 (0.78–0.86)0.76 (0.64–0.86)0.76 (0.68–0.83)0.76 (0.69–0.84)0.69 (0.62–0.76)
      RWS-HCCNR0.92 (0.88–0.95)
      10-year HCC risk model discrimination.
      NR0.84 (0.72–0.91)0.68 (0.57–0.79)0.67 (0.56–0.77)
      AASL-HCC0.81 (0.71–0.92)0.80 (0.72–0.89)0.85 (0.67–1.00)0.81 (0.67–0.94)0.73 (0.61–0.85)0.72 (0.64–0.80)
      NGM1-HCCNR0.85NR0.830.63 (0.53–0.73)0.56 (0.48–0.64)
      NGM2-HCCNR0.85NR0.830.67 (0.58–0.76)0.63 (0.55–0.72)
      CAMD0.82 (0.80–0.83)NR0.75 (0.72–0.77)0.76 (0.74–0.77)0.75 (0.67–0.83)0.72 (0.66–0.78)
      REAL-B0.81 (0.78–0.84)0.80 (0.78–0.83)0.83 (0.78–0.87)0.81 (0.77–0.85)0.76 (0.69–0.83)0.75 (0.70–0.81)
      AASL-HCC, age, sex, albumin, and liver cirrhosis for hepatocellular carcinoma; AUROC, area under the receiver-operating characteristic curve; BFH, Beijing Friendship Hospital; CAMD, cirrhosis, age, male sex, and diabetes mellitus; CI, confidence interval; CU-HCC, Chinese University-hepatocellular carcinoma; GAG-HCC, guide with age, gender, HBV DNA, core promoter mutations and cirrhosis-hepatocellular carcinoma; HCC, hepatocellular carcinoma; LSM-HCC, liver stiffness measurement-hepatocellular carcinoma; mPAGE-B, modified platelet, age, gender HBV; mREACH-B, modified risk estimation for hepatocellular carcinoma in chronic hepatitis B; NGM-HCC, nomogram-hepatocellular carcinoma; NR, not reported; PAGE-B, platelet, age, gender and HBV; mPAGE-B, modified platelet, age, gender and HBV; REACH-B, risk estimation for hepatocellular carcinoma in chronic hepatitis B; REAL-B, real-world effectiveness from the Asia-Pacific Rim for HBV risk score; RWS-HCC, real-world risk score for hepatocellular carcinoma.
      a 10-year HCC risk model discrimination.
      Subgroup analysis indicated similar results (Supplementary Appendix 9; Figure 2). The REAL-B exhibited best discrimination in nearly all subgroups for 3- and 5-year HCC prediction, in treated (0.83 and 0.81 for 3- and 5-year prediction, respectively), cirrhotic (0.70 and 0.66 for 3- and 5-year prediction, respectively), or noncirrhotic (0.76 and 0.75 for 3- and 5-year prediction, respectively) patients. In contrast, significant decrease of discrimination in the REACH-B was detected for treated vs untreated patients (P = .004 for 5-year prediction and P < .001 for 10-year prediction) and cirrhotic vs noncirrhotic patients (P = .003 for 3-year prediction). No significant difference of discrimination was found in other models across different subgroups; however, subgroup results by race might be not robust because of insufficient studies in non-Asians. Regarding calibration, underestimation of HCC risk was detected in the REACH-B (3-year total O:E ratio = 2.58) and PAGE-B (5-year total O:E ratio = 1.70) for cirrhotic patients, while the mPAGE-B overestimated risk in treated, cirrhotic, and noncirrhotic patients with a total O:E ratio ranging from 0.18 to 0.46.
      Figure thumbnail gr2
      Figure 2Discrimination of HCC risk prediction models in original derivation cohort, meta-analysis of external validation, and BFH validation cohort plus calibration plots in the BFH validation cohort. (A) Three-year HCC in the total BFH cohort; (B) 5-year HCC in the total BFH cohort; (C) 3-year HCC by cirrhosis; (D) 5-year HCC by cirrhosis. (E) Calibration plots for the CAMD, REAL-B, and mPAGE-B in the BFH validation cohort. Among all models, the REAL-B and CAMD showing best discrimination for treated patients within both the meta-analysis and BFH external validation, either in cirrhotic or noncirrhotic patients. In 4 models developed in untreated patients, GAG-HCC achieved best discrimination in treated patients. REAL-B and CAMD models exhibited best calibration whereas mPAGE-B overestimated HCC risk in the BFH external validation.

       Results of External Validation in the BFH Cohort

      Similarly, the REAL-B showed highest discrimination in the total BFH cohort (for 3-year prediction: 0.76; 95% CI, 0.69–0.83; for 5-year prediction: 0.75; 95% CI, 0.70–0.81), followed by CAMD (for 3-year prediction: 0.75; 95% CI, 0.67–0.83; for 5-year prediction: 0.72; 95% CI, 0.66–0.78). Generally, models developed in treated patients achieved higher discrimination ranging from 0.72 (PAGE-B: 95% CI, 0.56–0.89) to 0.76 (mPAGE-B: 95% CI, 0.69–0.84; REAL-B: 0.69–0.83) for 3-year prediction, whereas models developed in untreated patients showed lower AUROCs, ranging from 0.63 (NGM1-HCC: 95% CI, 0.53–0.73) to 0.68 (REACH-B: 95% CI, 0.51–0.85), except for GAG-HCC, achieving good discrimination (for 3-year prediction: 0.74; 95% CI, 0.62–0.87). A similar trend was shown in 5-year HCC risk prediction (Table 3 and Figure 2).
      Head-to-head comparison (Supplementary Appendix 10) indicated that the AUROC of the REAL-B was significantly higher than the REACH-B, NGM1-HCC, NGM2-HCC, CU-HCC, mREACH-BI, mREACH-BII, and mPAGE-B, while the CAMD had significantly higher AUROC than NGM1-HCC, NGM2-HCC, CU-HCC, mREACH-BI, PAGE-B, and RWS-HCC (all P values <.05). The GAG-HCC also tested a significantly higher AUROC than the NGM-1HCC, RWS-HCC, mREACH-BI, and PAGE-B (all P values <.05).
      Calibration validation was consistent with meta-analyzed results (Supplementary Appendix 11). Both the REAL-B and CAMD calibrated well in the BFH cohort, with 3-year BSs of 0.066 and 0.031, respectively, whereas the mPAGE-B overestimated HCC risk (BSs of 0.073 and 0.128 for 3- and 5-year prediction).
      Subgroup analysis on calibration demonstrated similar results (Supplementary Appendices 12 and 13). Among 14 models, the REAL-B and CAMD exhibited the best discrimination and calibration within subgroups of cirrhotic, noncirrhotic, elevated, or normal baseline ALT levels. Additionally, sensitivity analysis after multiple imputation also confirmed previous results (Supplementary Appendix 14).
      Overall, 9 models provided score cutoff points for clinical application, stratifying population to low, intermediate, or high risk (Table 1). As Table 4 shows, for either 3- or 5-year HCC prediction, most models achieved high sensitivity (around 90%) except for the GAG-HCC (for 3-year prediction: 80%; 95% CI, 65.6%–94.5%; for 5-year prediction: 68.5%; 95% CI, 55.6%–81.4%) and RWS-HCC (for 3-year prediction: 71.1%; 95% CI, 54.3%–87.9%; for 5-year prediction: 61.3%; 95% CI, 47.7%–74.8%), whereas nearly all models showed a specificity of approximately 30% with exception of GAG-HCC (for 3-year prediction: 65.2%; 95% CI, 61.5%–68.9%; for 5-year prediction: 71.4%; 95% CI, 64.7%–78.0%) and RWS-HCC (for 3-year prediction: 60.3%; 95% CI, 56.5%–64.0%; for 5-year prediction: 63.9%; 95% CI, 56.9%–70.9%). The GAG-HCC was observed to be most efficient, with 59% patients excluding from HCC risk in 3- and 5-year predictions (95% CI, 55%–62%), without any increase in failure rates across any models (Supplementary Appendix 15).
      Table 4Predictive Accuracy Measures With 95% CI of 9 Prediction Models Validated in the BFH Validation Cohort (n = 986)
      ModelCutoff PointSensitivity (%)Specificity (%)PPV (%)NPV (%)Efficiency (%)Failure Rate (%)
      3-y HCC occurrence
      GAG-HCC≥10080.0 (65.6–94.5)65.2 (61.5–68.9)7.6 (4.5–10.6)98.9 (98.1–99.8)58.5 (55.3–61.6)1.2 (0.5–2.6)
      CU-HCC≥586.6 (74.3–98.9)33.9 (30.3–37.6)4.4 (2.7–6.1)98.6 (97.3–100.0)26.5 (23.7–29.3)1.3 (0.4–3.9)
      LSM-HCC>1096.6 (90.0–100.0)19.7 (16.4–22.9)4.3 (2.7–6.0)99.4 (98.1–100.0)18.2 (15.4–20.6)0.7 (0.1–4.9)
      PAGE-B>1096.4 (89.3–100.0)18.6 (15.7–21.6)3.9 (2.4–5.3)99.3 (98.1–100.0)17.9 (15.5–20.5)0.7 (0.1–3.7)
      mPAGE-B>993.1 (83.9–100.0)36.4 (32.7–40.1)4.8 (3.0–6.6)99.4 (98.5–100.0)33.8 (30.8–36.9)0.7 (0.2–2.7)
      RWS-HCC≥4.571.1 (54.3–87.9)60.3 (56.5–64.0)5.9 (3.4–8.4)98.3 (97.2–99.5)34.9 (31.9–37.9)1.7 (0.7–4.1)
      AASL-HCC>596.9 (90.8–100.0)23.6 (20.3–26.8)4.2 (2.7–5.8)99.5 (98.6–100.0)21.1 (18.6–23.8)0.5 (0.1–3.5)
      CAMD≥886.2 (73.6–98.8)33.2 (29.7–36.8)4.2 (2.6–5.9)98.6 (97.2–100.0)27.1 (24.3–29.9)0.8 (0.2–3.1)
      REAL-B>396.8 (90.5–100.0)26.0 (22.6–29.4)4.3 (2.7–5.9)99.6 (98.7–100.0)23.1 (20.4–25.9)0.5 (0.1–2.7)
      5-y HCC occurrence
      GAG-HCC≥10068.5 (55.6–81.4)71.4 (64.7–78.0)16.0 (10.5–21.5)96.6 (94.9–98.3)58.5 (55.3–61.6)3.5 (2.1–5.8)
      CU-HCC≥587.1 (78.0–96.2)43.5 (36.2–50.8)10.5 (7.4–13.7)97.8 (96.1–99.5)26.5 (23.7–29.3)2.8 (1.3–6.2)
      LSM-HCC>1093.5 (86.3–100.0)24.7 (17.7–31.7)9.1 (6.4–11.8)97.9 (95.5–100.0)18.2 (15.4–20.6)2.6 (0.8–7.8)
      PAGE-B>1093.3 (85.9–100.0)20.7 (14.7–26.6)7.9 (5.7–10.2)97.7 (95.0–100.0)17.9 (15.5–20.5)2.6 (0.8–7.9)
      mPAGE-B>992.1 (84.6–99.6)42.5 (35.2–49.9)10.7 (7.5–13.8)98.6 (97.3–100.0)33.8 (30.8–36.9)1.6 (0.6–4.2)
      RWS-HCC≥4.561.3 (47.7–74.8)63.9 (56.9–70.9)11.7 (7.6–15.9)95.5 (93.4–97.5)34.9 (31.9–37.9)3.0 (1.5–5.9)
      AASL-HCC>594.4 (88.1–100.0)33.9 (26.9–40.9)10.0 (7.2–12.8)98.7 (97.3–100.0)21.1 (18.6–23.8)1.8 (0.6–5.6)
      CAMD≥887.2 (78.2–96.2)43.2 (36.0–50.3)10.5 (7.4–13.6)97.8 (96.1–99.4)27.1 (24.3–29.9)2.3 (1.0–5.5)
      REAL-B>394.0 (87.3–100.0)37.5 (30.3–44.7)10.1 (7.2–13.1)98.8 (97.4–100.0)23.1 (20.4–25.9)1.7 (0.6–5.3)
      NOTE. For the REACH-B, mREACH-BI, mREACH-BII, NGM1-HCC, and NGM2-HCC models, no cutoff value was provided in the original published paper.
      AASL-HCC, age, sex, albumin, and liver cirrhosis for hepatocellular carcinoma; BFH, Beijing Friendship Hospital; CAMD, cirrhosis, age, male sex, and diabetes mellitus; CI, confidence interval; CU-HCC, Chinese University-hepatocellular carcinoma; GAG-HCC, guide with age, gender, HBV DNA, core promoter mutations and cirrhosis-hepatocellular carcinoma; HCC, hepatocellular carcinoma; LSM-HCC, liver stiffness measurement-hepatocellular carcinoma; mPAGE-B, modified platelet, age, gender and HBV; mREACH-B, modified risk estimation for hepatocellular carcinoma in chronic hepatitis B; NGM-HCC, nomogram-hepatocellular carcinoma; NPV, negative predicted value; PAGE-B, platelet, age, gender and HBV; PPV, positive predicted value; REACH-B, risk estimation for hepatocellular carcinoma in chronic hepatitis B; REAL-B, real-world effectiveness from the Asia-Pacific Rim for HBV risk score; RWS-HCC, real-world risk score for hepatocellular carcinoma
      Efficiency = (True negative +False negative)/total cohort; Failure rate was calculated by Kaplan-Meier method in patients with low risk.

      Discussion

      We conducted a systematic review and meta-analysis of 14 previously developed HCC models in CHB patients with independent external validation of all models in the BFH cohort and compared their predictive performance. Overall, all models achieved acceptable discrimination both in meta-analysis and independent external validation. Specifically, 2 recently developed models, REAL-B and CAMD, showed best discrimination and calibration in treated patients, either in cirrhotic or noncirrhotic patients or in elevated/normal baseline ALT levels. However, the mPAGE-B seemed overestimating 3- and 5-year HCC risk while the REACH-B could underestimate 3- and 5-year HCC risk in cirrhotic patients undergoing antiviral therapy.
      To our knowledge, this is the first study to collect and aggregate all external validation studies for HCC risk prediction models in CHB patients, with comprehensive search for model identification and critical appraisal using the PROBAST checklist.
      • Wolff R.F.
      • Moons K.G.M.
      • Riley R.D.
      • et al.
      PROBAST: a tool to assess the risk of bias and applicability of prediction model studies.
      Moreover, we synthesized all evidence on each model performance at multiple available time points, and also conducted detailed subgroup analysis by important factors (treatment, cirrhosis, ALT level, and race) to address heterogeneity and robust of model performance. Additionally, for the first time, we assessed comparative performance across all these models with head-to-head validation in an independent, large sample size, prospective cohort of treated patients. Not only discrimination and calibration ability, but also efficiency and failure rate according to suggested cutoffs of each model were compared, which reflected the effect on daily clinical practice using these models.
      Generally, nearly all models
      • Yuen M.F.
      • Tanaka Y.
      • Fong D.Y.
      • et al.
      Independent risk factors and predictive score for the development of hepatocellular carcinoma in chronic hepatitis B.
      • Yang H.I.
      • Sherman Su J.
      • et al.
      Nomograms for risk of hepatocellular carcinoma in patients with chronic hepatitis B virus infection.
      • Wong V.W.
      • Chan S.L.
      • MO F.
      • et al.
      Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
      • Yang H.I.
      • Yuen M.F.
      • Chan H.L.
      • et al.
      Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
      • Wong G.L.
      • Chan H.L.
      • Wong C.K.
      • et al.
      Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B.
      • Lee H.W.
      • Yoo E.J.
      • Kim B.K.
      • et al.
      Prediction of development of liver-related events by transient elastography in hepatitis B patients with complete virological response on antiviral therapy.
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      • Poh Z.
      • Shen L.
      • Yang H.I.
      • et al.
      Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B.
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      contained age and gender as one of the predictors regardless of treatment status, whereas models developing in treated patients
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      ,
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      did not include HBV DNA, ALT, and HBeAg status as predictors. This might be the reason that these 3 factors could be greatly improved or altered during antiviral therapy in most patients, particularly with HBV DNA and ALT returning to be normal within 6 months.
      • Ono A.
      • Suzuki F.
      • Kawamura Y.
      • et al.
      Long-term continuous entecavir therapy in nucleos(t)ide-naïve chronic hepatitis B patients.
      • Kobashi H.
      • Takaguchi K.
      • Ikeda H.
      • et al.
      Efficacy and safety of entecavir in nucleoside-naive, chronic hepatitis B patients: phase II clinical study in Japan.
      • Kaneko S.
      • Kurosaki M.
      • Tamaki N.
      • et al.
      Tenofovir alafenamide for hepatitis B virus infection including switching therapy from tenofovir disoproxil fumarate.
      HBeAg status could also be altered after long-term therapy.
      • Koike K.
      • Suyama K.
      • Ito H.
      • et al.
      Randomized prospective study showing the non-inferiority of tenofovir to entecavir in treatment-naïve chronic hepatitis B patients.
      ,
      • Tamaki N.
      • Kurosaki M.
      • Kusakabe A.
      • et al.
      Hepatitis B surface antigen reduction by switching from long-term nucleoside/nucleotide analogue administration to pegylated interferon.
      As a result, they were not associated with HCC risk in patients undergoing antiviral therapy but only proved to be risk predictors in untreated patients. Therefore, in the era of antiviral therapy, untreated models (REACH-B and NGM-HCC) may not be suitable for HCC prediction in patients under antiviral treatment. In contrast, predictors (eg, PLT, ALB and cirrhosis) related to severity of fibrosis and cirrhosis were all included in treated models (CAMD, REAL-B, PAGE-B, mPAGE-B, AASL-HCC, and mREACH-B), meaning fibrosis and cirrhosis was the leading factor associated with HCC risk in treated patients, which was consistent with previous studies.
      • Papatheodoridis G.V.
      • Dalekos G.N.
      • Yurdaydin C.
      • et al.
      Incidence and predictors of hepatocellular carcinoma in Caucasian chronic hepatitis B patients receiving entecavir or tenofovir.
      ,
      • Kim M.N.
      • Hwang S.G.
      • Rim K.S.
      • et al.
      Validation of PAGE-B model in Asian chronic hepatitis B patients receiving entecavir or tenofovir.
      ,
      • Kirino S.
      • Tamaki N.
      • Kaneko S.
      • et al.
      Validation of hepatocellular carcinoma risk scores in Japanese chronic hepatitis B cohort receiving nucleot(s)ide analog.
      Furthermore, diabetes and alcohol intake were also included in treated models, which highlighted the need for clinicians to focus on these modifiable factors in order to further prevent fibrosis progression and minimize HCC risk.
      • Papatheodoridis G.V.
      • Voulgaris T.
      • Papatheodoridi M.
      • et al.
      Risk Scores for hepatocellular carcinoma in chronic Hepatitis B: a promise for precision medicine.
      Given better performance of models developed in treated patients, we recommend applying these treated models in daily clinical practice, particularly the CAMD and REAL-B, depending on availability of the predictors. The CAMD is much simpler than the REAL-B without the need for PLT and AFP testing. However, the cutoff validation result indicated that all treated models had lower efficiency (around 30%) with a comparable failure rate around 1%–3%. Thus, it would lead to nearly 70% of patients receiving HCC screening, which would be costly and cause extra stress to the patients. Therefore, it may be helpful to set higher cutoff values for these models, especially for the CAMD and REAL-B. Further model impact studies incorporating cost-effectiveness analysis and HCC screening intensity with participation rate are needed to validate the cutoff and corresponding net benefit.
      Consistent with previous studies,
      • Kim M.N.
      • Hwang S.G.
      • Rim K.S.
      • et al.
      Validation of PAGE-B model in Asian chronic hepatitis B patients receiving entecavir or tenofovir.
      ,
      • Seo Y.S.
      • Jang B.K.
      • Um S.H.
      • et al.
      Validation of risk prediction models for the development of HBV-related HCC: a retrospective multi-center 10-year follow-up cohort study.
      The PAGE-B did not show excellent discrimination performance compared with other scores (eg, REAL-B, CAMD, and AASL-HCC) in both meta-analysis and external validation. The reason for different predictive performance of the PAGE-B in Asian and Caucasian populations remains unclear. The different age at acquisition of HBV might be one of the reasons, as most Asians may acquire HBV at birth or early childhood due to endemic circumstances, while Caucasians mainly acquire HBV later in life.
      • Mittal S.
      • Kramer J.R.
      • Omino R.
      • et al.
      Role of age and race in the risk of hepatocellular carcinoma in veterans with hepatitis B virus infection.
      Thus, Asians may have longer duration of infection and higher HCC risk than Caucasians, which may ultimately affect model performance. Another explanation might be the difference of HBV genotypes, as genotypes B/C and A/D/G are separately common in Asia and Europe.
      • Liu C.J.
      • Kao J.H.
      Global perspective on the natural history of chronic hepatitis B: role of hepatitis B virus genotypes A to.
      • Cooksley W.G.
      Do we need to determine viral genotype in treating chronic hepatitis B?.
      • Kao J.H.
      • Chen P.J.
      • Lai M.Y.
      • et al.
      Hepatitis B genotypes correlate with clinical outcomes in patients with chronic hepatitis B.
      Thus, it may lead to different disease progression and therefore different model performance.
      Meanwhile, our study also suggested models such as the CAMD, REAL-B, and GAG-HCC, which heavily weight the presence of cirrhosis, achieved better predictive performance. That is particularly relevant, given that cirrhosis is strongly associated with HCC occurrence irrespective of treatment status.
      • Wong V.W.
      • Chan S.L.
      • MO F.
      • et al.
      Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
      ,
      • Abu-Amara M.
      • Cerocchi O.
      • Malhi G.
      • et al.
      The applicability of hepatocellular carcinoma risk prediction scores in a North American patient population with chronic hepatitis B infection.
      ,
      • Voulgaris T.
      • Papatheodoridi M.
      • Lampertico P.
      • et al.
      Clinical utility of hepatocellular carcinoma risk scores in chronic hepatitis B.
      However, accurate diagnosis of cirrhosis by histology through liver biopsy is not always available, and the diagnostic accuracy of serum panels, ultrasonography, or elastography is far from excellent. Therefore, it highlighted the need to accurately diagnose cirrhosis with uniform definition, particularly after antiviral therapy. Currently, according to several clinical guidelines,
      • Terrault N.A.
      • Lok A.S.
      • McMahon B.J.
      • et al.
      Update on prevention, diagnosis, and treatment of chronic hepatitis B: AASLD 2018 hepatitis B guidance.
      • Sarin S.K.
      • Kumar M.
      • Lau G.K.
      • et al.
      Asian-Pacific clinical practice guidelines on the management of hepatitis B: a 2015 update.
      • Lampertico P.
      • Agarwal K.
      • Berg T.
      • et al.
      EASL 2017 Clinical Practice Guidelines on the management of hepatitis B virus infection.
      for noncirrhotic patients, only those with annual HCC incidence >0.2% are recommended surveillance, while all cirrhotic patients should be under HCC surveillance due to their high risk. So, it is doubtful about the clinical utility of these models in cirrhotic patients. Further risk model studies should pay more attention to noncirrhotic patients.
      Additionally, only 6 of 14 models presented the full prediction model, such as all regression coefficients and baseline survival at a specific time point, to allow further individual prediction in clinical practice and subsequent external calibration validation.
      • Yang H.I.
      • Yuen M.F.
      • Chan H.L.
      • et al.
      Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
      ,
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      ,
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      • Hsu Y.C.
      • Yip T.C.
      • Ho H.J.
      • et al.
      Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
      • Yu J.H.
      • Suh Y.J.
      • Jin Y.J.
      • et al.
      Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
      • Yang H.I.
      • Yeh M.L.
      • Wong G.L.
      • et al.
      Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
      Only when a model achieves both good discrimination and calibration through broad external validation across different populations can it be considered for wide clinical use. Thus, it is urgent for researchers to present their prediction models substantially and transparently according to TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist.
      • Collins G.S.
      • Reitsma J.B.
      • Altman D.G.
      • et al.
      Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.
      Besides, both meta-analysis and independent validation showed that the mPAGE-B could overestimate HCC risk with acceptable discrimination, which may relate to the ethnicity issue (different HBV genotypes and age at acquisition of HBV) as the score was developed based on PAGE-B in Caucasian treated patients.
      • Papatheodoridis G.V.
      • Dalekos G.
      • Sypsa V.
      • et al.
      PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
      ,
      • Kim J.H.
      • Kim Y.D.
      • Lee M.
      • et al.
      Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
      Thus, it may not perform well in Asian populations and may need recalibration. However, overestimation of risk is more acceptable than underestimation of risk, as overestimation may lead to inefficiency of surveillance, but underestimation may lead to omitting patients who actually develop HCC and which thereafter threatens their life.
      However, several limitations of our analysis should be mentioned. First, diagnostic criteria of cirrhosis across different studies were not identical, with few studies using histology and most based on clinical diagnosis using ultrasonography or serological data. Thus, cirrhosis might be misclassified and further affect the performance of models containing cirrhosis as a predictor. Second, we did not have access to obtain individual patient data in each cohort, only studies reporting model performance measures of each subgroup were included in subgroup analysis, which could lead to insufficient sample size and lack of power of some subgroups (eg, different baseline ALT level and race subgroup). However, this might be the only way to guarantee internal validity of subgroup results and properly assessed potentially effect modifiers. Finally, because some models (ie, REAL-B, CAMD, and AASL-HCC) were recently published, few studies externally validated them until now. Thus, results should be interpreted with caution. However, external validation in the BFH cohort was consistent with meta-analysis, also indicating best performance for REAL-B and CAMD. Further external validation studies are needed to confirm this finding.

      Conclusions

      Of all 14 published HCC models, the REACH-B, CU-HCC, GAG-HCC, PAGE-B, and mPAGE-B were externally validated most across different populations. The REACH-B and NGM-HCC models performed relatively poor in discrimination, particularly in patients with cirrhosis and antiviral therapy. The REAL-B and CAMD models exhibited best discrimination and calibration within both meta-analysis and external validation. Calibration performance was poorly reported in most model development or external validation studies and is expected be improved in future studies involving prediction model. Future guidelines may incorporate these findings, taking into account also the implications in terms of model impact studies for assessing their potential clinical utility.

      Acknowledgments

      The authors thank all cooperating organizations and their staff whose hard work made this study possible. They thank Prof. Grace Lai–Hung Wong (The Chinese University of Hong Kong), Prof. Yao-Chun Hsu (Fu-Jen Catholic University), Prof. Terry Cheuk-Fung Yip (The Chinese University of Hong Kong), Prof. Beom Kyung Kim (Yonsei University College of Medicine), Prof. Seung Up Kim (Yonsei University College of Medicine), Prof. Mi Young Jeon (Yonsei University College of Medicine), Prof. Mi Na Kim (Yonsei University College of Medicine), Prof. Han Chu Lee (University of Ulsan College of Medicine), and Prof. Chee-Kiat Tan (Singapore General Hospital) for supplying observed/expected number of hepatocellular carcinoma events, area under the curve with 95% confidence interval, and/or data about subgroup model discrimination and calibration performance.

      Supplementary Material

      References

        • World Health Organization
        Global Hepatitis Report 2017.
        2017 (Available at:) (Accessed June 3, 2020)
        • Singal A.G.
        • El-Serag H.B.
        Hepatocellular carcinoma from epidemiology to prevention: translating knowledge into practice.
        Clin Gastroenterol Hepatol. 2015; 13: 2140-2151
        • World Health Organization
        Guidelines for the Prevention, Care and Treatment of Persons With Chronic Hepatitis B Infection.
        World Health Organization, Geneva, Switzerland2015
        • Cho J.Y.
        • Paik Y.H.
        • Sohn W.
        • et al.
        Patients with chronic hepatitis B treated with oral antiviral therapy retain a higher risk for HCC compared with patients with inactive stage disease.
        Gut. 2014; 63: 1943-1950
        • Terrault N.A.
        • Lok A.S.
        • McMahon B.J.
        • et al.
        Update on prevention, diagnosis, and treatment of chronic hepatitis B: AASLD 2018 hepatitis B guidance.
        Hepatology. 2018; 67: 1560-1599
        • Sarin S.K.
        • Kumar M.
        • Lau G.K.
        • et al.
        Asian-Pacific clinical practice guidelines on the management of hepatitis B: a 2015 update.
        Hepatol Int. 2016; 10: 1-98
        • Lampertico P.
        • Agarwal K.
        • Berg T.
        • et al.
        EASL 2017 Clinical Practice Guidelines on the management of hepatitis B virus infection.
        J Hepatol. 2017; 67: 370-398
        • Wong V.W.
        • Janssen H.L.
        Can we use HCC risk scores to individualize surveillance in chronic hepatitis B infection?.
        J Hepatol. 2015; 63: 722-732
        • Yuen M.F.
        • Tanaka Y.
        • Fong D.Y.
        • et al.
        Independent risk factors and predictive score for the development of hepatocellular carcinoma in chronic hepatitis B.
        J Hepatol. 2009; 50: 80-88
        • Yang H.I.
        • Sherman Su J.
        • et al.
        Nomograms for risk of hepatocellular carcinoma in patients with chronic hepatitis B virus infection.
        J Clin Oncol. 2010; 28: 2437-2444
        • Wong V.W.
        • Chan S.L.
        • MO F.
        • et al.
        Clinical scoring system to predict hepatocellular carcinoma in chronic hepatitis B carriers.
        J Clin Oncol. 2010; 28: 1660-1665
        • Yang H.I.
        • Yuen M.F.
        • Chan H.L.
        • et al.
        Risk estimation for hepatocellular carcinoma in chronic hepatitis B (REACH-B): development and validation of a predictive score.
        Lancet Oncol. 2011; 12: 568-574
        • Wong G.L.
        • Chan H.L.
        • Wong C.K.
        • et al.
        Liver stiffness-based optimization of hepatocellular carcinoma risk score in patients with chronic hepatitis B.
        J Hepatol. 2014; 60: 339-345
        • Lee H.W.
        • Yoo E.J.
        • Kim B.K.
        • et al.
        Prediction of development of liver-related events by transient elastography in hepatitis B patients with complete virological response on antiviral therapy.
        Am J Gastroenterol. 2014; 109: 1241-1249
        • Papatheodoridis G.V.
        • Dalekos G.
        • Sypsa V.
        • et al.
        PAGE-B: A risk score for hepatocellular carcinoma in Caucasians with chronic hepatitis B under a 5-year entecavir or tenofovir therapy.
        J Hepatol. 2016; 64: 800-806
        • Poh Z.
        • Shen L.
        • Yang H.I.
        • et al.
        Real-world risk score for hepatocellular carcinoma (RWS-HCC): a clinically practical risk predictor for HCC in chronic hepatitis B.
        Gut. 2016; 65: 887-888
        • Kim J.H.
        • Kim Y.D.
        • Lee M.
        • et al.
        Modified PAGE-B score predicts the risk of hepatocellular carcinoma in Asians with chronic hepatitis B on antiviral therapy.
        J Hepatol. 2018; 69: 1066-1073
        • Hsu Y.C.
        • Yip T.C.
        • Ho H.J.
        • et al.
        Development of a scoring system to predict hepatocellular carcinoma in Asians on antivirals for chronic hepatitis B.
        J Hepatol. 2018; 69: 278-285
        • Yu J.H.
        • Suh Y.J.
        • Jin Y.J.
        • et al.
        Prediction model for hepatocellular carcinoma risk in treatment-naive chronic hepatitis B patients receiving entecavir/tenofovir.
        Eur J Gastroenterol Hepatol. 2019; 31: 865-872
        • Yang H.I.
        • Yeh M.L.
        • Wong G.L.
        • et al.
        Real-world effectiveness from the Asia Pacific Rim liver consortium for HBV risk score for the prediction of hepatocellular carcinoma in chronic hepatitis B patients treated with oral antiviral therapy.
        J Infect Dis. 2020; 221: 389-399
        • Coffin C.S.
        • Rezaeeaval M.
        • Pang J.X.
        • et al.
        The incidence of hepatocellular carcinoma is reduced in patients with chronic hepatitis B on long-term nucleos(t)ide analogue therapy.
        Aliment Pharmacol Ther. 2014; 40: 1262-1269
        • Arends P.
        • Sonneveld M.J.
        • Zoutendijk R.
        • et al.
        Entecavir treatment does not eliminate the risk of hepatocellular carcinoma in chronic hepatitis B: limited role for risk scores in Caucasians.
        Gut. 2015; 64: 1289-1295
        • Kim W.R.
        • Loomba R.
        • Berg T.
        • et al.
        Impact of long-term tenofovir disoproxil fumarate on incidence of hepatocellular carcinoma in patients with chronic hepatitis B.
        Cancer. 2015; 121: 3631-3638
        • Ahn J.
        • Lim J.K.
        • Lee H.M.
        • et al.
        Lower observed hepatocellular carcinoma incidence in chronic hepatitis B patients treated with entecavir: results of the ENUMERATE study.
        Am J Gastroenterol. 2016; 111: 1297-1304
        • Moons K.G.M.
        • de Groot J.A.H.
        • Bouwmeester W.
        • et al.
        Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.
        PLoS Med. 2014; 11e1001744
        • Wolff R.F.
        • Moons K.G.M.
        • Riley R.D.
        • et al.
        PROBAST: a tool to assess the risk of bias and applicability of prediction model studies.
        Ann Intern Med. 2019; 170: 51-58
        • Wu S.
        • Kong Y.
        • Piao H.
        • et al.
        On-treatment changes of liver stiffness at week 26 could predict 2-year clinical outcomes in HBV-related compensated cirrhosis.
        Liver Int. 2018; 38: 1045-1054
        • Bruix J.
        • Sherman M.
        • American Association for the Study of Liver Diseases
        Management of hepatocellular carcinoma: an update.
        Hepatology. 2011; 53: 1020-1022
        • Debray T.P.
        • Damen J.A.
        • Riley R.D.
        • et al.
        A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes.
        Stat Methods Med Res. 2019; 28: 2768-2786
        • Wong G.L.
        • Chan H.L.
        • Chan H.Y.
        • et al.
        Accuracy of risk scores for patients with chronic hepatitis B receiving entecavir treatment.
        Gastroenterology. 2013; 144: 933-944
        • Chen W.
        • Chen J.
        • Zheng Q.
        • et al.
        Validation study of prediction models of hepatitis B virus-related hepatocellular carcinoma.
        Zhonghua Gan Zang Bing Za Zhi. 2015; 23: 507-511
        • Kim G.A.
        • Lee H.C.
        • Kim M.J.
        • et al.
        Incidence of hepatocellular carcinoma after HBsAg seroclearance in chronic hepatitis B patients: a need for surveillance.
        J Hepatol. 2015; 62: 1092-1099
        • Abu-Amara M.
        • Cerocchi O.
        • Malhi G.
        • et al.
        The applicability of hepatocellular carcinoma risk prediction scores in a North American patient population with chronic hepatitis B infection.
        Gut. 2016; 65: 1347-1358
        • Tawada A.
        • Chiba T.
        • Saito T.
        • et al.
        Utility of prediction scores for hepatocellular carcinoma in patients with chronic hepatitis B treated with nucleos(t)ide analogues.
        Oncology. 2016; 90: 199-208
        • Papatheodoridis G.V.
        • Dalekos G.N.
        • Yurdaydin C.
        • et al.
        Incidence and predictors of hepatocellular carcinoma in Caucasian chronic hepatitis B patients receiving entecavir or tenofovir.
        J Hepatol. 2015; 62: 363-370
        • Kim M.N.
        • Hwang S.G.
        • Rim K.S.
        • et al.
        Validation of PAGE-B model in Asian chronic hepatitis B patients receiving entecavir or tenofovir.
        Liver Int. 2017; 37: 1788-1795
        • Yang H.I.
        • Tseng T.C.
        • Liu J.
        • et al.
        Incorporating serum level of hepatitis b surface antigen or omitting level of hepatitis B virus DNA does not affect calculation of risk for hepatocellular carcinoma in patients without cirrhosis.
        Clin Gastroenterol Hepatol. 2016; 14: 461-468.e2
        • Brouwer W.P.
        • van der Meer A.J.P.
        • Boonstra A.
        • et al.
        Prediction of long-term clinical outcome in a diverse chronic hepatitis B population: Role of the PAGE-B score.
        J Viral Hepat. 2017; 24: 1023-1031
        • Daheim M.
        • Lang S.
        • Goeser T.
        • et al.
        Real-world risk score for hepatocellular carcinoma risk prediction in CHBV: a validation outside of Asia.
        Gut. 2017; 66: 1346-1347
        • Riveiro-Barciela M.
        • Tabernero D.
        • Calleja J.L.
        • et al.
        Effectiveness and safety of entecavir or tenofovir in a Spanish cohort of chronic hepatitis B Patients: validation of the Page-B score to predict hepatocellular carcinoma.
        Dig Dis Sci. 2017; 62: 784-793
        • Seo Y.S.
        • Jang B.K.
        • Um S.H.
        • et al.
        Validation of risk prediction models for the development of HBV-related HCC: a retrospective multi-center 10-year follow-up cohort study.
        Oncotarget. 2017; 8: 113213-113224
        • Jeon M.Y.
        • Lee H.W.
        • Kim S.U.
        • et al.
        Feasibility of dynamic risk prediction for hepatocellular carcinoma development in patients with chronic hepatitis B.
        Liver Int. 2018; 38: 676-686
        • Yip T.C.
        • Wong G.L.
        • Wong V.W.
        • et al.
        Reassessing the accuracy of PAGE-B-related scores to predict hepatocellular carcinoma development in patients with chronic hepatitis B.
        J Hepatol. 2020; 72: 847-854
        • Ono A.
        • Suzuki F.
        • Kawamura Y.
        • et al.
        Long-term continuous entecavir therapy in nucleos(t)ide-naïve chronic hepatitis B patients.
        J Hepatol. 2012; 57: 508-514
        • Kobashi H.
        • Takaguchi K.
        • Ikeda H.
        • et al.
        Efficacy and safety of entecavir in nucleoside-naive, chronic hepatitis B patients: phase II clinical study in Japan.
        J Gastroenterol Hepatol. 2009; 24: 255-261
        • Kaneko S.
        • Kurosaki M.
        • Tamaki N.
        • et al.
        Tenofovir alafenamide for hepatitis B virus infection including switching therapy from tenofovir disoproxil fumarate.
        J Gastroenterol Hepatol. 2019; 34: 2004-2010
        • Koike K.
        • Suyama K.
        • Ito H.
        • et al.
        Randomized prospective study showing the non-inferiority of tenofovir to entecavir in treatment-naïve chronic hepatitis B patients.
        Hepatol Res. 2018; 48: 59-68
        • Tamaki N.
        • Kurosaki M.
        • Kusakabe A.
        • et al.
        Hepatitis B surface antigen reduction by switching from long-term nucleoside/nucleotide analogue administration to pegylated interferon.
        J Viral Hepat. 2017; 24: 672-678
        • Kirino S.
        • Tamaki N.
        • Kaneko S.
        • et al.
        Validation of hepatocellular carcinoma risk scores in Japanese chronic hepatitis B cohort receiving nucleot(s)ide analog.
        J Gastroenterol Hepatol. 2020; 35: 1595-1601
        • Papatheodoridis G.V.
        • Voulgaris T.
        • Papatheodoridi M.
        • et al.
        Risk Scores for hepatocellular carcinoma in chronic Hepatitis B: a promise for precision medicine.
        Hepatology. 2020; 72: 2197-2205
        • Mittal S.
        • Kramer J.R.
        • Omino R.
        • et al.
        Role of age and race in the risk of hepatocellular carcinoma in veterans with hepatitis B virus infection.
        Clin Gastroenterol Hepatol. 2018; 16: 252-259
        • Liu C.J.
        • Kao J.H.
        Global perspective on the natural history of chronic hepatitis B: role of hepatitis B virus genotypes A to.
        J Semin Liver Dis. 2013; 33: 97-102
        • Cooksley W.G.
        Do we need to determine viral genotype in treating chronic hepatitis B?.
        J Viral Hepat. 2010; 17: 601-610
        • Kao J.H.
        • Chen P.J.
        • Lai M.Y.
        • et al.
        Hepatitis B genotypes correlate with clinical outcomes in patients with chronic hepatitis B.
        Gastroenterology. 2000; 118: 554-559
        • Voulgaris T.
        • Papatheodoridi M.
        • Lampertico P.
        • et al.
        Clinical utility of hepatocellular carcinoma risk scores in chronic hepatitis B.
        Liver Int. 2020; 40: 484-495
        • Collins G.S.
        • Reitsma J.B.
        • Altman D.G.
        • et al.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.
        BMJ. 2015; 350: g7594