Impact of Regulatory Approval Status on CADTH Reimbursement of Oncology Drugs and Role of Real-World Evidence on Conditional Approvals from 2019 to 2021

Round 1

Reviewer 1 Report

Summary

The current manuscript under review is an interesting addition to a growing body of work around the impacts of real-world evidence (RWE) on regulatory decision-making and the health technology assessment (HTA) process in Canada. Looking at reimbursement recommendations from Canada’s main HTA agency, CADTH, between 2019 and 2021, authors examine how RWE has been used in decision making, and the impact of RWE on CADTH’s final recommendations on conditional approvals. The authors note a main strength being that this is the first comprehensive review of how regulatory decisions influenced reimbursement decisions in Canada for recently approved oncology drugs, and the role of RWE in decision-making processes. While authors found that regulatory approval status influences reimbursement outcomes for oncology drugs in Canada, little insight about robustness of RWE needed for favorable considerations was gleaned. This article appears especially suited for the special edition under review.

 

Specific Comments

It is not clear to this reviewer why the specific timeframe of 2019-2021 was used, and not a broader timeframe. It is notable that the majority of the timeline examined was during the COVID-19 pandemic, yet there is no comment about the pandemic or it’s potential impact on the number/type of HTA submissions being made during this event. 

It was also noted that there is no comment on the inclusion of RWE in submissions beyond NOCc – it would be interesting to see for example if the NOCc status drives the inclusion of RWE, or if sponsors have been regularly including RWE in their submissions more broadly. Was this examined at all during the current work?

I note that Table 1 and Table 2 examine in detail review timelines @ Health Canada and CADTH for the 2019-2021 products, but did not notice a direct comparison to previously published timelines in the literature which I think would be valuable – are the timelines in the current study consistent/shorter/longer than previous timelines, and if different any potential reasons for there (capacity/volume increases or decreases; pandemic impacts etc)

I found Table 3 difficult to read/follow in it’s current format, but recognize that design editing will likely make this easier to follow. I support the authors’ choice of examples to include in this table. Table 3 title feels a bit confusing in current format, consider something more in line with the textual description of this table like:   Examples of criticisms from CADTH appraisals of NOCc approvals

I think more attention needs to be given to the conclusion that “RWE needs to be improved”; that there is no information available at this time from CADTH about what specifically would get the improvements they want to see is key

Overall, it feels to this reader like the broader conclusion from this analysis is that there is currently no demonstrable role for RWE regarding conditional approvals, despite there be (loudly) growing consensus about the need for RWE in this space. That not only do we need “better” RWE, but we need a better and more transparent commitment to the generation and use of RWE in this space perhaps.

Author Response

The authors would like to thank reviewer #1 for the interest in the manuscript and the support for importance of good quality RWE be used for regulatory and reimbursement submissions.

To address Reviewer’s Specific Comments

1. It is not clear to this reviewer why the specific timeframe of 2019-2021 was used, and not a broader timeframe. It is notable that the majority of the timeline examined was during the COVID-19 pandemic, yet there is no comment about the pandemic or it’s potential impact on the number/type of HTA submissions being made during this event. 

Response to #1 :

In response to the reviewer’s comment on the specific time-frame 2019-2021 selected for review, the reasons are listed below:

Timing of publication of Health Canada and CADTH guidance document for use of RWE in regulatory decision making: The first publication acknowledging that RWE could potentially be used for regulatory decision was published in 2016 (ref 4, line 45), the preliminary guidance document (“Elements of Real World Data/Evidence Quality throughout the Prescription Drug Product Life Cycle”) was published in 2019 (ref 5, line 50) and a strategy document published in March 2020 spelled out how Health Canada in collaboration with the Canadian Agency for Drugs and Technology in Health (CADTH) intended to operationalize the incorporation of RWD/E into decision making (ref 6, line 63). Although some forms of RWE were used in regulatory submissions before 2019, the time period was selected to capture RWE submissions during and after offical publication of guidance documents. The following sentence was added to line 236:

“ As the guidance document was published by Health Canada in 2019 and the collaboration with CADTH to operationalize the incorporation of RWE into decision making was announced in 2020 , the time period 2019-2021 was selected to optimally capture submissions that would have incorporated the elements of the guidance document.”

Previous publications coverage: Two previous publications (ref 32 line 243 and ref 35 line 263) covered funding situations on oncology drugs by CADTH for period 2010-2017 and 2011-2019 respetively. The focuses of these previous publications were somewhat different but general concerns regarding funding of oncology drugs in Canada were the common theme.

A sentence was added in line 263 regarding the period studied in ref 35 was from 2011-2019. The authors noticed reference 35 was mistakenly inserted so a correction was made. (line 478).

Impact of COVID 19 on regulatory submission:

Before Covid 19 pandemic, Health Canada was going through a period of regulatory reform known as R2D2 (regulatory reforms for drugs and devices) and that prepared them well for Covid 19 additional regulatory burden. Health Canada announced at the beginning of the pandemic that an agile regulatory system allowed Covid 19 vaccines and treatments to be approved via Interim Orders, thus lessening regulatory burden without compromising safety and efficiacy evaluations. The progressive work format adopted by Health Canada allowed them to achieve approvals of new non-covid drugs and devices without much interruptions. A sentence was added in line 250 to mention that performance of Health Canada was not impacted by Covid 19 pandemic.

“The Covid 19 pandemic did not appear to have impacted Health Canada oncology drug approval performance as the number of submissions/approvals from 2019-2021 was similar to 2017-2018 as assessed from reviewing Health Canada Drug Submission Performance Annual reports (direct request from publications-publications@hc-sc.gc.ca).”  

2. It was also noted that there is no comment on the inclusion of RWE in submissions beyond NOCc – it would be interesting to see for example if the NOCc status drives the inclusion of RWE, or if sponsors have been regularly including RWE in their submissions more broadly. Was this examined at all during the current work?

Response to #2:

This is an excellent question. Real World Data/Evidence is included in most CADTH submissions as CADTH guidelines required Canadian Comparators for construction of economic models (line 271-274) especially when Canadian comparators were not performed within the pivotal phase 3 trials. Unlike NOCc submissions which included mainly small, single arm trials, both standard and priority submissions (NOCs) included randomized clinical trials (RCTs) in their submissions. As mentioned in line 275, with the availability of placebo- or comparator- data from the RCTs, “anchored MAIC” in a network meta-analysis environment can potentially be conducted generating results based on more reliable data-platforms. The authors’ comments were based on review of NOCs with positive and negative CADTH recommendations on NOC submissions. For example, for Zanubrutinib (+ve reimbursement recommendation), clinical reviews included positive comments on ITC and an anchored MAIC: “The ITC was informed by an appropriately conducted systematic review of the literature highlighting the relevant population and outcomes of interest for this review” and “ A comprehensive list of prognostic factors and treatment-effect modifiers identified through appropriate channels was included in the report and — based on discussions with the clinical experts consulted by CADTH — these factors and modifiers were considered relevant,”.

For a -ve reimbursement recommendation such as for Sonidegib, clinical review statements included “the sponsor submitted a published unanchored MAIC comparing the two treatments in this patient population. Two trials were included: the BOLT trial which provided IPD for sonidegib, and the ERIVANCE trial which provided aggregate data for treatment with vismodegib. No statistical comparisons between the treatments were provided and minimal adjustment for potential effect modifiers and prognostic factors was provided. Further, no assessment of residual confounding was performed. As such, no conclusions can be made regarding the comparative efficacy of sonidegib and vismodegib based on the submitted unanchored MAIC”  

When comparing CADTH’s critical appraisals of sponsor submitted RWE, differentiation between +ve and -ve decisions were generally science based for standard submissions. However for NOCc, CADTH’s critical appraisals of sponsor-submitted RWE were similar for +ve or -ve decisions, thus failing to provide insight as to level of evidence required for successful reimbursements.

A sentence “A selected review of oncology drugs with NOC approvals showed CADTH critical appraisals of RWE differentiated positive from negative reimbursement decisions” (line 285) and three references (41-43) were added to provide more information regarding CADTH’s feedback for standard submissions.

3. I note that Table 1 and Table 2 examine in detail review timelines @ Health Canada and CADTH for the 2019-2021 products, but did not notice a direct comparison to previously published timelines in the literature which I think would be valuable – are the timelines in the current study consistent/shorter/longer than previous timelines, and if different any potential reasons for there (capacity/volume increases or decreases; pandemic impacts etc)

Response to #3

The authors searched the literature and could not locate publications providing historical “time in review” for Health Canada or CADTH for comparison purpose. Health Canada drug approval performance reports from 2016 were requested from Publication department of Health Canada from 2014 to 2020. The overall new drug submissions from 2015-2019 were 62, 62, 53, 80 and 73 and new drug approvals from 2016-2019 were 35, 56, 61 and 77 respectively. The big jump between 2017 and 2018 was due to regulatory reforms and improved efficiencies, thus increasing the capacity to accept more new drugs for regulatory reviews. Due to the big jump for 2018, in order address reviewer’s question of impact of pandemic on review capacity and volume, the best comparisons would be comparing drug-approval performance between 2019-2021 to 2017-2018. When enumerated as oncology drugs approved between 2017-2018 by Health Canada, 30 new approvals were found, and this compared very well to the 45 oncology drug approved between 2019-2021. As mentioned in response to comment #1, Health Canada went through a regulatory reform before the pandemic and was operating under an agile regulatory mandate during the pandemic, so performance was not affected. Sponsors in general submit to CADTH shortly before or after regulatory approvals so number of submissions received by CADTH would also be similar between the 2 time periods mentioned above.

For clarification, a sentence of comparisons between 2017-2018 to 2019-2021 was inserted in line 250.

“The Covid 19 pandemic did not appear to have impacted Health Canada oncology drug approval performance as the number of submissions/approvals from 2019-2021 was similar to 2017-2018 as assessed from reviewing Health Canada Drug Submission Performance Annual reports (direct request from publications-publications@hc-sc.gc.ca)” 

4. I found Table 3 difficult to read/follow in it’s current format, but recognize that design editing will likely make this easier to follow. I support the authors’ choice of examples to include in this table. Table 3 title feels a bit confusing in current format, consider something more in line with the textual description of this table like:   Examples of criticisms from CADTH appraisals of NOCc approvals

Response to #4:

The authors agree with the reviewer that the format of Table 3 could potentially be confusing to the author and the Title can be modified to reflect the content better, so both the title and format of Table 3 has been modified and will be replaced by the revised format in the revised manuscript.

5. I think more attention needs to be given to the conclusion that “RWE needs to be improved”; that there is no information available at this time from CADTH about what specifically would get the improvements they want to see is key

Response to #5

The authors agree with the reviewer and the entire conclusion has been re-written to incorporate reviewer’s suggestions:

“The findings of the current paper revealed that NOCc oncology drugs recently approved by Health Canada had the highest rate of negative reimbursement recommendations by pCODR. The results also suggested that RWE was used to a limited extent in supporting the data gaps. In cases where RWE was part of the submission, data quality issues did not ameliorate the uncertainty in the evidence package.

Therefore the quality of RWE needs to be improved and one approach would be to create national standards for data quality. This maybe achieved through the development of RWE consensus guidelines created by a key collaboration of all the key stakeholders. Until such time, the quality of RWE generated to support NOCc drug submissions will remain suboptimal and the full potential of such a data source will never be realized.”

6. Overall, it feels to this reader like the broader conclusion from this analysis is that there is currently no demonstrable role for RWE regarding conditional approvals, despite there be (loudly) growing consensus about the need for RWE in this space. That not only do we need “better” RWE, but we need a better and more transparent commitment to the generation and use of RWE in this space perhaps.

Response to #6:

We are in full agreement with the reviewer.  It really is a “chicken and egg” situation.  On the one hand, our findings suggest the role of RWE in conditional approvals has been limited to date.  On the other hand, there is indeed a growing consensus about the need for more RWE in this space.  In our view, it comes down to data quality.  If better quality RWE were to become available, it would be more credible with pCODR reviewers and the expert committee pERC.  These group would then become more comfortable is using such information during their review of products with NOCc. We once again revert to our recommendation on the development of national standards and RWE consensus guidelines.

Reviewer 2 Report

1. In the first table in the Supplementary materials, it would be helpful (if possible) to add tumor type and Health Canada approval type.

2. While perhaps beyond he scope of this paper, it would be of interest if the authors were able to add:

a. How much of a price reduction did CADTH suggest for agents approved with conditions?

b. How long after the CADTH review were negotiations finalized with pCPA (if ever)

c. For drugs not approved by CADTH, which ones have been approved by other countries such as US, UK, Germany, France?

3.  Does the first author have an affiliation (job or current school)?

4. Reference #35 is incorrect. 

Author Response

1. In the first table in the Supplementary materials, it would be helpful (if possible) to add tumor type and Health Canada approval type.

Response to #1:

The authors agree with the addition of tumor types to the Supplemental Tables. Tables S1-3 have been revised with tumor types added for each drug listed. As for Health Canada approval types, they were listed in the titles. Table S1 for Standard Approvals, Table S2 for Conditional Approvals and Table S3 for Priority Approvals.

2. While perhaps beyond the scope of this paper, it would be of interest if the authors were able to add:
3. How much of a price reduction did CADTH suggest for agents approved with conditions?

Response to 2a: Cost issues were beyond the scope of the paper.  However, we are considering a second paper looking a cost and cost effectiveness

1. How long after the CADTH review were negotiations finalized with pCPA (if ever)

Response to 2b: This is also beyond scope of this manuscript

1. For drugs not approved by CADTH, which ones have been approved by other countries such as US, UK, Germany, France?

Response to 2c: The authors agree that while the comparisons of CADTH decisions to international reimbursement recommendations would be of interest, it is however beyond the scope of this manuscript as our focus is Canadian reimbursement decisions which have direct impact on Canadian patients’ ability to access cancer treatments.

3. Does the first author have an affiliation (job or current school)?

Response to 3: First Author is an independent Consultant and has no affiliation with industry or academia.

4. Reference #35 is incorrect. 

Response to 4: Ref #35 has been rectified in the revised manuscript

Reviewer 3 Report

It is not clear why the Cancer drugs from 2019-2021 were chosen for this study. Would it be possible for the authors to explain?

Author Response

In response to the reviewer’s comment on the specific time-frame 2019-2021 selected for review, the reasons are listed below:

Timing of publication of Health Canada and CADTH guidance document for use of RWE in regulatory decision making: The first publication acknowledging that RWE could potentially be used for regulatory decision was published in 2016 (ref 4, line 45), the preliminary guidance document (“Elements of Real World Data/Evidence Quality throughout the Prescription Drug Product Life Cycle”) was published in 2019 (ref 5, line 50) and a strategy document published in March 2020 spelled out how Health Canada in collaboration with the Canadian Agency for Drugs and Technology in Health (CADTH) intended to operationalize the incorporation of RWD/E into decision making (ref 6, line 63). Although some forms of RWE were used in regulatory submissions before 2019, the time period was selected to capture RWE submissions during and after offical publication of guidance documents. The following sentence was added to line 236:

“ As the guidance document was published by Health Canada in 2019 and the collaboration with CADTH to operationalize the incorporation of RWE into decision making was announced in 2020 , the time period 2019-2021 was selected to optimally capture submissions that would have incorporated the elements of the guidance document.”

Previous publications coverage: Two previous publications (ref 32 line 243 and ref 35 line 263) covered funding situations on oncology drugs by CADTH for period 2010-2017 and 2011-2019 respetively. The focuses of these previous publications were somewhat different but general concerns regarding funding of oncology drugs in Canada were the common theme.