TECH-VER: A Verification Checklist to Reduce Errors in Models and Improve Their Credibility

Background In health economic literature, checklists or best practice recommendations on model validation/credibility always declare verifcation of the programmed model as a fundamental step, such as ‘is the model implemented correctly and does the implementation accurately represent the conceptual model?’ However, to date, little operational guidance for the model verifcation process has been given. In this study, we aimed to create an operational checklist for model users or reviewers to verify the technical implementation of health economic decision analytical models and document their verifcation eforts. Methods Literature on model validation, verifcation, programming errors and credibility was reviewed systematically from scientifc databases. An initial beta version of the checklist was developed based on the checklists/tests identifed from the literature and from authors’ previous modeling/appraisal experience. Next, the frst draft checklist was presented to a number of health economists on several occasions and was tested on diferent models (built in diferent software, developed by diferent stakeholders, including drug manufacturers, consultancies or academia), each time leading to an update of the checklist and culminating in the fnal version of the TECHnical VERifcation (TECH-VER) checklist, introduced in this paper. Results The TECH-VER necessitates a model reviewer (preferably independent), an executable and transparent model, its input sources, and detailed documentation (e.g. technical report/scientifc paper) in which the conceptual model, its implementation, programmed model inputs, and results are reported. The TECH-VER checklist consists of fve domains: (1) input calculations; (2) event-state (patient fow) calculations; (3) result calculations; (4) uncertainty analysis calculations; and (5) other overall checks (e.g. validity or interface). The frst four domains refect the verifcation of the components of a typical health economic model. For these domains, as a prerequisite of verifcation tests, the reviewer should identify the relevant calculations in the electronic model and assess the provided justifcations for the methods used in the identifed calculations. For this purpose, we recommend completeness/consistency checks. Afterwards, the verifcation tests can be conducted for the calculations in each of these stages by checking the correctness of the implementation of these calculations. For this purpose, the following type of tests are recommended in consecutive order: (i) black-box tests, i.e. checking if model calculations are in line with a priori expectations; (ii) white-box testing, i.e. going through the program code details line by line, or cell by cell (recommended for some crucial calculations and if there are some unexpected results from the black-box tests); and (iii) model replication/parallel programming (recommended only in certain situations, and if the issues related to the identifed unexpected results from black-box tests could not be resolved through white-box testing). To reduce the time burden of model verifcation, we suggest a hierarchical order in tests i–iii, where going to the next step is necessary when the previous step fails. Conclusions The TECH-VER checklist is a comprehensive checklist for the technical verifcation of decision analytical models, aiming to help identify model implementation errors and their root causes while improving the transparency and efciency of the verifcation eforts. In addition to external reviews, we consider that the TECH-VER can be used as an internal training and quality control tool for new health economists, while developing their initial models. It is the authors’ aim that the TECH-VER checklist transforms itself to an open-source living document, with possible future versions, or ‘bolt-on’ extensions for specifc applications with additional ‘ft-for-purpose’ tests, as well as ‘tips and tricks’ and some demonstrative error examples. For this reason, the TECH-VER checklist and the list of black-box tests created in this paper and a few model verifcation examples is uploaded to an open access, online platform (github and the website of the institute), where other users will also be able to upload their original verifcation eforts and tests.

• View at Publisher

Free Full Text ( Final Version , 1mb )