In exercises designed to evaluate the potential magnitude of bias of the IHE Diabetes Cohort Model, we compared results generated for a set of simulation scenarios with those of a micro-simulation model (Economic and Health Outcomes Model of T2DM), chosen because the structures are otherwise generally similar and because it was possible to harmonize the models even more to minimize between-model simulation differences. We found systematic differences in simulated costs and quality-adjusted life-years, but little evidence of systematic differences in the incremental costs and quality-adjusted life-years that underlie cost-effectiveness metrics or in incremental cost-effectiveness ratios and net monetary benefits themselves.

 

Discussion

Using well-established cross-validation tools [11] modified to allow structural standardization of the models, we examined whether IHE-DCM produces systematically biased estimates of cost-effectiveness related to the cohort approach. In a simple Reference Case performed to enable comparison with the results of 11 other models that participated in the 9th Mount Hood Diabetes Challenge, IHE-DCM produced consistently greater absolute survival, QALYs, and costs than ECHO-T2DM, which is consistent with the difference between modeling homogenous patients and heterogeneous patients when event risks are non-linear (specifically convex) in key parameters [16]. Between-model differences were generally small at the incremental level (i.e., different between the two comparator arms) used to construct cost-effectiveness metrics, however, and the ICER and NMB, which were also similar between models. As expected, IHE-DCM was considerably faster compared with ECHO-T2DM, with a run time of approximately 45 min compared with 30 h using ECHO-T2DM, an important aspect for many stakeholders under time constraints.

This same pattern was observed for the more realistic Expanded Reference Case and 18 scenario analyses, and both models responded to changes in model parameters similarly and predictably. This was supported statistically; incremental costs, incremental QALYs, and NMBs for each model fell uniformly within the 95% confidence interval generated by the other model. There was more uncertainty in the results of IHE-DCM, which was driven in large part by uncertainty in the parameter estimate for the hypoglycemia event rate (eliminating it roughly halved the confidence interval). The estimates of ECHO-T2DM falls within even half of the 95% confidence intervals generated by IHE-DCM. Estimates in the base case by IHE-DCM of the 40-year cumulative incidence of study outcomes, moreover, fell within the 95% confidence intervals generated by ECHO-T2DM. While the paired t tests did find statistically significant between-model differences in incremental costs, incremental QALYs, and the NMB for these 19 scenarios, the paired t test is grossly overpowered to reject the null hypothesis in this setting as the simulation scenarios (i.e., the sample draws) are not independent of each other. Interestingly, however, the paired t test failed to reject between-model differences for the ICER (p < 0.68) for the 18 scenarios for which both incremental costs and incremental QALYs were positive (producing a meaningful ICER). Further underscoring this absence of clear bias in cost-effectiveness estimates, there was no discernible pattern as to which model produced more favorable cost-effectiveness estimates, with each more favorable in roughly half of the scenarios.

The trade-off between cohort modeling and micro-simulation is sometimes (perhaps mistakenly) cast as a choice between time and transparency vs accuracy. Both models satisfy International Society for Pharmacoeconomics and Outcomes Research recommendations for model transparency, which accept complexity and call instead for a technical report that describes the structure, components, equations, and computer code that would enable experts to reproduce the model (full technical transparency) and non-technical documentation that, at a minimum, describes the type of model and intended applications, funding sources, model structure, inputs and outputs, data sources, model validation, and model limitations [11]. While transparency in a general sense is hard to quantify, and no fit-for-purpose diabetes models are likely to achieve “transparency” in a general sense, analysts (authors AN and AL) generally considered that IHE-DCM was easier to grasp and work with (and is constructed with approximately 50% fewer lines of code).

This analysis has several strengths, including the use of two models that were relatively similar and required limited standardization. Many of the remaining differences could be standardized to minimize the extent that differences would be driven by model differences other than units of observations. The scenarios were inspired by the Mount Hood Reference Case, which permits comparison (at least of the Reference Case results) with 11 health economic models of diabetes that participated in the 9th Mount Hood Diabetes Challenge Network. Finally, a wide range of scenarios was considered that explored different aspects of the model to enhance generalizability.

The models could not be entirely standardized, however, and remaining differences must be considered when interpreting the results of this analysis (i.e., between-model differences may reflect more than just the potential bias related to cohort vs micro-simulation modeling). In particular, the main structural difference is the modeling of CKD, for which there are different methods of simulating disease progression (transition probability vs biomarker driven) and which clearly impact the results. Indeed, for the cumulative incidence, the CKD outcomes (micro-and macroalbuminuria and end-stage renal disease) were clear outliers and the mean estimates for the IHE-DCM model were just within the 95% confidence interval of ECHO-T2DM. Foot ulcer is included only in ECHO-T2DM. To mitigate the impact on the analysis, costs and QALY weights were set to 0. The indirect impact on overall results was limited because foot ulcer affected only the risk of congestive heart failure (though patients simulated to develop congestive heart failure had in turn increased risks for ischemic heart disease, myocardial infarction, and mortality) and the simulated incidence of foot ulcer was low. Second, while the scenarios were constructed to mimic a cost-effectiveness analysis, the simulated scenarios are purely hypothetical.

While this study cannot provide a definitive (and universal) answer to concerns about possible bias, and it does not address the academic discussion of how much accuracy is reasonable to swap for increased transparency [30], this exercise provides a careful examination of how two otherwise similar models respond to the same set of stimuli (both absolutely and incrementally), which can be valuable for stakeholders charged with interpreting evidence produced by IHE-DCM.

Conclusions

The IHE-DCM was faster to load and to run than the micro-simulation model used in this study (ECHO-T2DM) and the modeling details are likely to be more easily understood by external reviewers, which can be an advantage for economic stakeholders with limited time and resources. Despite systematic differences in absolute predicted survival, QALYs, and costs, estimated cost-effectiveness metrics were similar suggesting that any bias related to the cohort approach is small in the outcomes that matter most. We believe that both models are suitable for use in cost-effectiveness evaluations for interventions in T2DM; the selection of one over the other should be made on the basis of stakeholder needs, resources, and preferences.

 

Nyhetsinfo

www red DiabetologNytt