Introduction

The Institute for Clinical and Economic Review (ICER) is committed to open and transparent engagement with all stakeholders that have an interest in each of its evidence reviews.  This commitment to transparency extends to the development and/or modification of economic models. Such transparency helps to increase the public’s confidence in model results. Without detailed descriptions of model structure and processes as well as estimates used, economic models run the risk of being considered “black boxes,” with no way to evaluate the validity of model processes or accuracy of model inputs. Explicit delineation of model structure and flow gives stakeholders the ability to evaluate the model’s face validity. Details on the point estimates and ranges used in sensitivity analyses allow for the explicit testing of alternative assumptions and model inputs, provide insight into the drivers of specific results, and allow other interested parties to replicate or extend analyses conducted by ICER and its collaborators.

General Approach

Our general approach to model transparency is based on the Modeling Good Research Practices Task Force report on “Model Transparency and Validation” jointly produced by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision-Making (SMDM).[1]  Our aim is to provide information on the model structure and processes, all major inputs and sources for data, and key assumptions used in our economic analyses, so that readers can judge their confidence in the results while preserving the intellectual property rights of those we collaborate with.

All model documents will note that funding for ICER’s analyses is unrestricted and publicly disclosed.  In addition, ICER develops economic models in collaboration with academic researchers who are free from financial conflicts on any given project.  In addition, ICER maintains a strict conflict-of-interest policy for its own employees.

Policy

For each report, the interventions selected for study will be specified in detail. ICER and its collaborators will provide model documentation, including model structure, comparators, and specifications. When existing models are being used, ICER will provide references to prior publications that provide further details on the model. When new models are developed, this information will be provided as part of the technical report.

Following the publication of a final scope for each topic, ICER and its external collaborators will publish a modeling analysis plan with detailed specifications for the expected conduct of the work.  The plan will be published on a public website used to share collaborative research known as the Open Science Framework (https://osf.io/7awvd/), approximately 15 weeks after a topic is publicly announced (19 weeks for class reviews).  Stakeholders will be notified when the analysis plan is posted.  The plan may be updated following review of additional data sources, discussions with stakeholders, and other activities, and so is intended to be considered a “living document.” Detailed elements of the analysis plan will include:

  • Analytic objectives
  • Model structure, including a textual and/or graphic depiction of the model structure, process, and outputs
  • Descriptions of interventions and comparators
  • Perspective (generally health care system)
  • Time horizon (generally lifetime)
  • Discount rate
  • Key assumptions to be used in the model
  • Model input values, ranges, and sources of data
  • Other variables crucial to understanding model transition and flow (e.g., risk equations for downstream events)

Sources for model inputs, risk equations, etc. will be provided as part of the documentation. In general, ICER’s analyses will use data sources and information from published or publicly available sources, including peer-reviewed journals, supplementary appendices, briefing documents used by regulatory authorities, and conference proceedings. In specific instances, valid analyses may require the use of unpublished information, such as manufacturers’ data on file. In such circumstances, explicit requests will be made to affected parties, and any reasonable documentation to protect patient and/or stakeholder confidentiality will be provided. The final version of the modeling analysis plan will be used in conducting the ICER’s “long-term value for money” analyses. Importantly, the modeling analysis plan is intended to provide enough information for an experienced researcher to be able to replicate the economic model and analyses.

Importantly, the modeling analysis plan is intended to provide enough information for an experienced researcher to be able to replicate the economic model and analyses. Actual executable models and associated computer code will not be provided as part of the deliverable, as such an effort would unduly compromise the intellectual property rights of ICER’s external collaborators. As the ISPOR-SMDM Task Force has pointed out, without such protections, “the incentives and resources to build and maintain complex models could disappear.”1

Additionally, ICER and its collaborators will provide a summary of the results of these analyses in a model technical summary. This 10- to 15-page summary will be part of a larger report that ICER will produce that will include information on the available clinical evidence, current guidelines and payer coverage policies, and other relevant topics.  The model summary will consist of the following sections:

  • Methods, including key assumptions and key model inputs
    1. Overview, including description of model structure
    2. Perspective
    3. Patient Population
    4. Costs
    5. Quality of Life/Utility
    6. Primary, Alternative, and Sensitivity Analyses
    7. Budget Impact Analysis
    8. Appendices, including other assumptions and model inputs
  • Results
    1. Primary (Base-Case) Analysis Results
    2. Alternative and Sensitivity Analysis Results, including tornado diagram
    3. Budget Impact Analysis Results
    4. Appendices, including supporting tables/figures summarized in main text
  • Summary and Comment, including limitations and comparison to other published models on the topic of interest

The model results become sections of an ICER-published report on the comparative clinical effectiveness, cost-effectiveness and budget impact of the specific interventions being evaluated. The initial draft report will be posted for a public comment period of four weeks, after which it may be revised. 

At the outset of each project, ICER will gauge manufacturer interest in receiving an executable version of the draft cost-effectiveness model during the public comment period on the initial draft report.  When there is sufficient interest in a model sharing effort, manufacturers of the interventions and comparators of interest for a given review will pay a fee to ICER’s academic collaborators to cover the time needed to produce sharable executable files and all supporting documentation.  No fees will be assessed for models produced by internal ICER staff.  To protect the intellectual property of the researchers and their institutions, models will be shared under confidentiality or licensing agreements. Under these agreements, manufacturers will be unable to alter the model for their own purposes. In addition, their access will be for a limited timeframe and will be confined to the team of individuals responsible for the review. The expectation is that such a review will inform the written public comments on the draft report.

ICER sometimes uses academic- or commercial-in-confidence data provided by manufacturers in its economic models.  ICER’s policy on accepting these data can be found here. These data will be redacted from models shared through the transparency program.  However, manufacturers who provide such confidential data should be aware that it may be possible in some cases for model recipients to “back-calculate” these inputs.  To mitigate this risk, ICER and its economic modeling collaborators will include language in all licensing agreements that expressly prohibits the back-calculation of model inputs through “what-if” queries or other methods, and that restricts the ability of model recipients to save or reproduce the model itself.  Licensing agreements will also stipulate that organizations receiving the model must agree to keep any proprietary information included in the model confidential, including the model and any non-public data it contains.

The revised draft report is then presented as part of a public assessment meeting. The modeling sections of the report are intended to provide enough information to evaluate the economic analysis, but not necessarily all of the information that would be required to replicate the analysis.

ICER endeavors to follow recommended best practices throughout our evaluations. By following the process outlined above, we hope to make our economic models and associated analyses more transparent and useful to the health care community.

[1] Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, on behalf of the ISPOR−SMDM Modeling Good Research Practices Task Force. Model transparency and validation: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health 2012;15:843-850.