Considering Clinical, Real-World, and Unpublished Evidence
Comparative clinical effectiveness involves weighing the benefits and harms/burdens of one treatment option versus another. The most important benefits and harms are those that are important to patients and their families/caregivers. As such, from the outset of a review ICER solicits input from patients, families, caregivers, and expert clinicians to understand the day-to-day experience of living with a condition and what outcomes it would be most important for a therapy to affect. Information on what has been learned from patient input is presented in the ICER report prior to the discussions of the evidence so that readers can interpret the evidence through the lens of patient experience.
Stakeholder input from clinicians, manufacturers, and payers is used in addition to input from patients and families to frame the questions that an ICER comparative effectiveness review attempts to answer. When evidence on patient-important outcomes is limited or unavailable, ICER will seek evidence on surrogate endpoints that might be associated with outcomes important to patients and families.
ICER’s evaluation of comparative clinical effectiveness is grounded in a systematic review of all available evidence. A systematic review identifies all relevant existing evidence using explicit, replicable methods in a way that minimizes the risk of biased selection of studies. Established best methods of systematic literature reviews are followed in order to foster transparency and facilitate reproduction of results.
- ICER and Aetion Partner to Develop Real-World Evidence for Value Assessment of Treatments
- Policy on Inclusion of Grey Literature in Evidence Reviews
- Guide to Understanding Health Technology Assessment (HTA)
- ICER’s Methods for Health Technology Assessment
- ICER’s Commitment to Economic Model Transparency
- Guidelines on ICER’s Acceptance and Use of “In-Confidence” Data from Manufacturers of Pharmaceuticals, Devices, and other Health Interventions
Clinical Evidence
ICER’s judgements around comparative clinical effectiveness are informed by evidence arising from multiple sources. When available, high-quality RCTs or their meta-analyses provide evidence that is least susceptible to certain scientific biases. When benefits and harms occur over the course of many years, or when harms are rare but clinically important or even catastrophic, evidence from high quality published peer-reviewed studies using observational data and methodologies such as cohort studies, case-control studies, and long-term disease and drug registries may be used. Furthermore, if important patient reported outcomes have not been collected as part of a manufacturer’s clinical development program, ICER will again conduct a comprehensive literature review to identify published, peer-reviewed observational studies providing this information.
Real-World Evidence
RWE may help complement other types of evidence in assessments of comparative clinical effectiveness, in contributing to assessment of the potential other benefits of interventions, and in providing useful information to inform the assumptions of economic models. ICER has consistently sought to incorporate analysis of RWE into our reports whenever it can provide additional perspective on comparative clinical effectiveness or cost-effectiveness. In addition to searching for published RWE and seeking RWE in the grey literature, on several occasions we have collaborated with patient and other stakeholder organizations to obtain new patient and caregiver survey information when it was not available in the medical literature. Findings from this work have been included in our Evidence Reports and helped inform discussions during our Public Advisory meetings and appraisal committee votes.
RWE often has greater vulnerability to known and unknown biases that create limitations in our ability to rely on it when making judgments about relative effectiveness of different care options. Nonetheless, we understand that RCTs have their own limitations and are often inadequate to address all questions relevant to assessments of comparative clinical effectiveness. RWE can be particularly helpful under certain circumstances such as when long-term safety of a treatment or durability of a medication’s effect is unclear. We have also emphasized how RWE can be helpful in supporting consideration of a treatment’s “potential other benefits” that lie outside traditional clinical trials. Patient-reported outcome studies and studies that capture broader patient and family effects of treatment are especially desired as they can provide evidence usually not included in clinical trials.
Unpublished Evidence
ICER also includes evidence from the “grey literature.” We supplement our reviews of studies from peer-reviewed publications with data from
conference proceedings, regulatory documents, materials from other HTA groups, information submitted by manufacturers, and input gleaned from patients. Consideration of multiple sources of evidence helps us evaluate whether there is a biased representation of study results in the published literature and provides a panoramic understanding of net health benefit.
In summary, ICER has a flexible and inclusive approach to sources of evidence, which stresses the importance of the rigor of clinical trial while augmenting such evidence with data from other real-world or grey-literature sources.
All of this evidence feeds into our final conclusions on the certainty and magnitude of benefit for each health intervention, which is ultimately summarized by the rating we include in our reports from ICER’s evidence rating matrix. Finally, we also monitor new publications so we can determine if there is enough new evidence to update an existing assessment.
