Reg Confirmatory Adaptive Trials David Laurie, Michael Branson, Frank Bretz, Willi Maurer and Peter Thomas report on a European Medicines Agency workshop on the demands associated with designing and performing confirmatory adaptive clinical trials. Adaptive tral design (ATD), the clinical tral methodology that allows trial design modifications to be made during the course of a study, without compromising the scientific method, were the subject ofa lively workshop in London recently. Representatives from health agencies, industry, academia and patient groups took par in the meeting held on 14 December 2007 atthe European Medicines Agency, which looked atthe opportunities and challenges in the design and conduct of confirmatory adaptive clinical tials. ‘The mecting mainly covered Key concems that were raised in the recently published reflection paper from the EMEA's scientific advisory body, the Committee for Medicinal Products for Human Use (CHMP), on methodological issues in confirmatory clinical tials planned with an adaptive design (CHMP/EWP/2459/02) Further discussion will be arranged in a series of ‘meetings to be created by the EMEA\s scientific advice working party Introduction and overview Armin Koch, of Germany's Federal Institute for Drugs and Medical Devices (BEAM), reviewed the considerations behind the CHMP reflection paper. Dr Koch confirmed that its intention is to highlight potential issues, and advise prospective and thorough planning of ATDs. Regulators acknowledge that, even after Phase Il, there remain. uncertainties, and that ATDs can improve the speed and efficiency of clinical development; however, this should be achieved without lowering regulatory standards. ATDs are especially welcome if they improve the basis for regulatory decision-making, They can be appropriate in situations of “difficult reality” but not to rescue a poorly designed study. However, not all possible types of adaptation are acceptable, nor too many design modifications, and conclusions must be based on the terminal analysis with control of type I error. Importantly, sponsors must justify combinability of stages by assessing the homogeneity of data obtained before and after the interim analysis, Judith Quinan of GlaxoSmithKline gave a review of industry opinion. Industry's interest in ATDs is not only for speedier development, but also for improving the development process, resulting in better dose, benefit and tisk determination, and more effective and ethical stewardship of resources, including pationt data. Although uncertainty is reduced during the learning phase of development, there is still uncertainty at the start of the confirmatory phase. By “blurring” the boundaries between the classical Phases I and Il, or between Phases II and III, ATDs allow the ‘management of such uncertainty, especially in multinational confirmatory studies where there is a large commitment of resources. ‘Nevertheless, Dr Quinlan agreed wholeheartedly with Dr Koch that ATDs need thorough upfront planning and are not an excuse for poor planning. Admittedly, experience is limited. In the future, with more experience itis hoped that ATDs may become increasingly acceptable in the confirmatory phase. Moving forward, industry is keen to work with regulators on learning from. shared examples. Robert Hemmings of the UK's Medicines and Healthcare products Regulatory Agency, recommended consultations with the EMEA and described experiences from the review of many |ATDs submitted for scientific advice (15-20 in 2006/7). These were mostly Phase II/IIl ATDs on. dose selection and/or sample size, and were mostly for stand-alone pivotal studies. While some trial designs did not raise major concerns, most were disappointing and poorly planned (too frequent assessments; lack of type I error control; potential introduction of bias; and inappropriate adaptation). Importantly, Mr Hemmings advised sponsors to provide a rationale as to why the ATD provides a scientifically better outcome; not just savings of time or money. He added that its critical, to consider the totality of evidence and whether this will be adequate from a single pivotal ATD. At ‘Davi Uae PAD (Repaltony Policy Exper, Micha Branson PAD, CS (ical Head Siatinial Methodology) Fank Beets PRD (Glomete! Fellow) ana Will Manrer PD (ties, frmery hand of Sata Methoclogy) are t Novarte Pharma AG, Bae, ‘Srrtaetnd Petr Taomas Bi, CS (SelorManogee Methodology and Imgvaton) it Novarie Pharma Ud, Horsham, UK. The ‘ctor hk Beaece Obed ol Dr Regulatory Altits, Novars Phare AG, fo hr ei of hs are ‘The meeting covered Jey concerns about confirmatory trials, aac ised in a reeent paper by the EMEA‘s scientific advisory committee Adaptive trial designs can improve the speed cand eficiency of clinical development. but they require thorough upfront planning € Informa UK Lid 2008, ‘wwcraipharma.com AL Phorma Feb 2008When planning, it should be considered ‘fata are available elsewhere Learning can occur in ATDs in exploratory and confirmatory phases bu! ita a different context in these to setings Adaptive seamless designs can be accepted in the right situations eas present, since limited experience exists with ATDs, regulatory advice must be cautious and few ATD designs have been endorsed to date. However, regulators are not averse to properly planned and justified ATDs that adequately take account of risks, as outlined in the reflection. paper. SueJane Wang of the Office of Biostatistics, US Food and Drug Administration, provided some details of the agency’s experience with ATDs. A Key point is that confirmatory ATD studies should still meet the requirement of “adequate and well-controlled”, Adaptations should be limited, otherwise the principle of confirmatory trials is undermined, making them exploratory. When planning an ATD, it should be considered if data are available elsewhere (eg other marketed products in the same class) and outcome simulations are recommended. In study conduct, integrity is critical, especially data monitoring committee confidentiality arrangements. Dr Wang also advised that there should be a prospectively planned analysis on the consistency of results to assess the impact of the interim analyses. As with the EMEA, consultations with the FDA are recommended. In the subsequent discussion, the regulatory presenters elaborated on some situations of ‘difficult reality” where ATDs could be appropriate (eg rare diseases, unknown/variable placebo responsein certain diseases, dose-finding), and commented that it may depend upon the therapeutic area, Although dose selection can be improved, not too many doses should be investigated, and 'ATDs should not reduce focus on pharmacokinetic /pharmacodynamic investigations in Phase In response to the presentation from Dr Quinlan, Dr Koch did not see ATDs as blurring the distinction between learning and confirming, although he acknowledged that ATDs can resolve some uncertainties remaining prior to the confirmatory phase. In his opinion, Phase II/IITATDs are a particular challenge as there is an inherent conflict between learning and confirmation of hypotheses. ‘This led to a discussion on the nature of learning in confirmatory studies (eg learning about the appropriate dose or secondary endpoints). According to Dr Wang, although learning can occur in ATDs in both exploratory and confirmatory phases, it has a different context in these two settings. ‘Thus, further discussion on the context and terminology of “learning” (discovery, exploration, hypotheses confirmation, etc) in exploratory and confirmatory phases may be required. Phase II/III study combinations Both Mr Hemmings and Vlad Dragalin, of Wyeth, gave a formal definition of designs based on an article by Maca eal (2006), A seamless design operationally combines into single trial objectives ‘which are traditionally addressed in separate trials (such designs are thus operationally seamless). In adaptive seamless designs, on the other hand, the final analysis will use data for inference from. patients enrolled before and after the adaptation (such designs are thus inferentially seamless). Dr Dragalin presented some advantages of confirmatory adaptive designs from an industry perspective: faster drug development, lower costs, better use of data and greater certainty about the selected dose or doses. Mr Hemmings questioned some of the claimed benefits, such as the removal of the “white space” between Phases Il and Ill He asked if thinking/ consultation time was thereby lost and if timelines could even be extended by more complex upfront planning, logistics and regulatory interactions. He reiterated that the rationale for an adaptive design needs to justify ‘why sufficient evidence is expected from the Phase II/III combination trial compared to separate trials, and if there is an information (especially safety information) loss or increase on a project level due to the conduct of an adaptive design. He concluded that adaptive seamless designs ‘can be accepted in the right situations (persuasive rationale; helping better decision-making; and ‘methodologically sound). There was discussion later that the white space between studies is not just thinking time but is used for, among other things, protocol preparations, contract negotiations, ‘ethics committee reviews and clinical tral applications. Thus, performing these activities once for an ATD rather than twice for separate studies can save time. Case studies Andrew Stone of AstraZeneca and Frank Bretz of Novartis presented case studies, which illustrated how to conduct a dose selection at interim analysis while providing confirmatory evidence of efficacy at the final analysis, The AstraZeneca case study compared two dose levels with a control using Bayesian predictive power methodology to guide the interim decision. The Novartis case study involved the comparison of four dose levels against three (one placebo and two active) control groups for three hierarchically ordered clinical endpoints (see Figure 1). It was clear in the subsequent discussion that a strict type I error control is mandatory. As pointed out by Dr Koch, experimental hypotheses must be well specified upfront for any confirmatory study. Moreover, the number of open questions should be kept to a minimum ~ explorative, hypotheses-generating adaptive designs are not acceptable for confirmatory trials. Both case studies emphasised that careful preplanning is important, including the Feb 2000 RAS Pharma mmm ww zajphorme.com © Informa UK Ltd 2008cr ecatcoe Maker specification of the study objectives, response variables and statistical analysis strategies. The decision rules allowing early stopping for success have to be stated explicitly in the protocol. For other adaptations, like treatment selection or dropping a dose, this is not the case, since the type | error is not affected by the nature of these rules if appropriate statistical methods are used. The treatment selection atthe interim decision point can thus be based on any prior evidence, including, information from outside the ongoing trial. There was general agreement for extensive simulations in order to assess the impact of the interim decision rules on treatment effect estimates and the ‘operation characteristics ofthe tral, and as guidance for the decision-makers at interim analyses. ‘The Novartis case study also illustrated how flexbility might be implemented by proposing. potential interim decision rules in the data monitoring committee charter, while allowing the committee to deviate from them and contact the sponsor as necessary. Jn the discussion, Dr Wang and Mr Hemmings commented that, in most regulatory applications for approval, two pivotal trials are typically needed to replicate findings, though in some special circumstances a single adaptive Phase II/IIl study may suffice. The Novartis case study, in which a separate, non-adaptive study that would be started at the interim analysis of the adaptive study, could be an appropriate way of obtaining confirmation. In addition, Dr Wang, explained that the same preparatory information is required for Phase II/III ATDs as for normal ‘Phase III studies, except for the single open question, such as dose selection. Figure 1. Phase ill Adaptive Design with Treatment Selection: A case study trial design presented by Frank Bretz, Nove STAGE 1 (Phase'ib) STAGE 2 Phase ti) Placebo ‘Active control Active control2 ‘Active control2 Screening | Doseranging {interim} Efficacy and safety 1 2weeks analysis 26 weeks Sponsor involvement in adaptive trials Peter Bauer, of the University of Vienna, Austria, opened the session by relating his experience as ‘a member of an interim analysis decision board that reviewed clinical efficacy, selected doses and estimated the sample size for the subsequent stage of a study. ‘The interim analysis was carried out by an independent statistician who made a proposal to the decision board, comprised of external and company experts. Professor Bauer was convinced that the confidentiality of the decision was kept within the decision board, which is supported by the timely publication of the negative outcome of the trial Michael Krams of Wyeth gave a presentation that was jointly authored with Paul Gallo ‘of Novartis and based largely on the article of Gallo ef al (2006. A sponsor’s concern is that "unanticipated complexities might not fit a pre-specified algorithm that can be implemented without sponsor participation. Important sponsor interests may be involved and sponsor perspective may be relevant in arriving at the best decision. ‘A general model was proposed for limited sponsor involvement in confirmatory trials based oon the following principles: © Informe UK Lid 2008, ‘wort zajpharma.com In most regulatory ‘applications for approval, ‘00 pivotal studies ‘are typically needed to repeat findings ‘The case study compared {four dose levels against ‘three control groups {for three hierarchically ‘ordered clinical endpoints A general navel was proposed for tnt Sponsor inclement i confirmatory trial based om fe principles RAS Pharma Feb 2008 87Regulatory Feature Akey concern is maintaining tial ner eo lig ‘There isa risk that Heterogeneity in the results across stages may render the overall resulls hard to interpret... and a staged procedure for investigating this was proposed clear rationale for involvement; individuals properly distanced from trial operations; clear understanding by all involved of the potential risks; documentation of the processes followed and restrictive firewalls in place; and ‘minimal sponsor exposure sufficient to make decisions ~ ie by the smallest number of individuals, only at the adaptation point, and with the minimally relevant information. Dr Krams felt that itis possible to carefully control access to interim information in a manner that ‘maintains trial integrity and meets the needs of sponsors and regulators, ‘The current regulatory views on sponsor involvement in data monitoring committees were presented by Mr Hemmings, Regulators discourage sponsor involvement in interim analyses but understand the arguments for their involvement (financial implications, complexity of decision-making and the difficulty of formulating. a decision algorithm for independent experts in advance). A key concern is maintaining trial integrity ~ if the treatment allocation or accumulating, results are known or inferred, could the conduct of the trial be influenced or biased and how ccan the absence of bias be established? Mr Hemmings said protection of trial integrity and statistical validity are generally well done in present-day submissions, citing stopping rules, data monitoring committee charters, independent statisticians and firewalls. In his concluding, remarks, he warned that bias due to dissemination of information is not quantifiable and plans to protect information are not guaranteed to be sufficient. He questioned whether the benefits ‘of sponsor involvement outweigh the risks and challenged applicants to justify why sponsor involvement is necessary. During the discussion, the impact of information leakage, whether real or suspected, was raised, especially the relative impact compared to other, external events (eg public release of data ‘on closely related compounds). Mr Hemmings advised that a second confirmatory trial may lessen. the concern of sponsor involvement in a Phase II/III combination trial. He felt uncomfortable about decisions being made purely by prespecified algorithm, since all relevant scientific expertise should be available on the data monitoring committee. Markku Toivonen of NDA Regulatory Science Ltd (ex-CHMP) added that, if the sponsor is, involved, it must be for some additional value. In conclusion, Mr Hemmings repeated the advice ‘conceming sponsor involvement that confidence must be built by good examples, and that there are risks. Assessing change/establishing that the trial is reliable ‘A session at the workshop covered combining results from different stages of an ATD. Will Maurer of Novartis Pharma reviewed key points of the CHMP reflection paper, resultant issues and hurdles, and a high-level proposal on how to address them. The paper expresses concern that ‘heterogeneity in the results across stages may render the overall results hard to interpret, because it cannot be excluded that this difference is due to intentional or unintentional communication of interim results. The sponsor is, therefore, asked to pre-plan methods to ensure that the results from different stages can be justifiably combined. Dr Maurer agreed that the issue of heterogeneity should not be ignored, but stated that the problem is how to assess heterogeneity, what standards to apply and what to do if itis found, since it can exist even in non-adaptive studies and there is a risk of invalidation of genuine results, He questioned the reflection paper’s request for at least the same careful investigation of heterogeneity as usually required for meta analyses, because differences in effect size between trials included in a post hoc meta analysis are much more likely than differences between stages ‘within a well-planned adaptive trial. He stated that there are no conventional standards for acceptable homogeneity in other contexts and considered possible formal statistical investigations of homogeneity and their possible impact on the interpretation ofthe trial results. Since a standard approach (eg testing for heterogeneity ata significance level of 15%) would lead toa disturbingly high false positive signal rate, he asked if such an inital signal for heterogeneity should not be dependent on the observed overall effect strength Dr Maurer concurred that the investigation of heterogeneity of results between stages is as important as any other investigation of heterogeneity within a trial and proposed a staged procedure for investigating it One of the stages, namely the investigation ofa time trend unrelated to the adaptation of the design, was further investigated by Tim Friede of the University of Warwick in the UK, Feb 2008 RAJ Pharma mmm: wew-tajpharma.com © Informa UK Lis 2000Regulatory Feature ‘Tablo 1. Summary of EMEA mocting on confirmatory adaptive tl "14 December 2007" ‘Open questions ‘Session inroguctionovervaw Cost of R&D isa public health issue Growing patient pressure to make new medicines available Interim analysis ean be ethical and potentially “mandatory” ‘ATDs are not a remedy for poor planning Some uncertainty remains in Phase I CContzol of typeI ert s feasible and necessary Early stopping can reduce totality of evidence available Saving RED costs is a main driver for ATDs. [Need to save time since the totality of data is important ‘Complete blursing of exploratory and ‘confirmatory phases* (Chu the authors! opinion, although stated as disagreement, the contention relat to interpretation of terms, eg learinghreducton of uncertainty / exploration hypothesis testing. Further discussions may be needed) Do ATDs bring better information than traditional strategies? ‘Are ATDs only for “difcult reality"? Can interim analyses be used for conditional approval submissions? Isa single adaptive pivotal tril sufficient? —caseby-case basis Selection of (biomarker defined) target population? How to determine the success of ATD ? Seamless desion ‘Some reassuring examples of seamless sign exist There isa need for replication to gain more experience What are the real benefits of seamless designs? How much benefit from long-term fllow= up of Phase I patients? ‘Sponcorinolvenent “Thal integrity isthe primary goal of all stakeholders Risk of operational basis obvious Sponsor involvement isnot impossble but “Limited sponsor involvement should be the rule ‘What are “unanticipated complexities” in decision-making? ~ examples are needed “How to “prove” absence of bias (probabilistic approach) ust be justified | Timing of paediatec studies Heterogenely "ATDs mean improved transparency “How to quantify (and tolerate) Price to pay: multiplicity and incecsed | heterogeneity | warenees of heterogeneity I “How toasters information leakage ro Indy oar eed work ot [ * Published with the permission of Bruno Flamion. The euthor ofthis rte have added some small lrfctions ad comment, Professor Friede presented results regarding operation characteristics of a “usual” test of — The meeting included heterogeneity under various assumptions (no heterogeneity, presence of time trend) and concluded discussions of seamless that it leads to great loss of power if coupled with an invalidation of a statistically significant — design, sponsors, treatment effect. He also showed that it cannot be compensated for by larger sample sizes, and that heterogeneity and future calendar time effects unrelated to interim analyses make matters worse. Alternative approaches steps allowing for calendar time effects need more attention. ‘Keaven Anderson of Merck Research Laboratories presented four examples of trials where heterogeneous results over the trial duration may have had various causes, such as change of treatment guidance, dose selection, enrolment trends and just random fluctuations. Dr Anderson stated that inference concerning the global null hypothesis nevertheless should not be an issue and that heterogeneity has often not prohibited trials from being confirmatory or approvable. He asked, given that heterogeneity in treatment effect among sequential subgroups can be large due {to random variation, would the expectation of homogeneity in an adaptive trial not create a double standard compared to other designs? Dr Koch, of BEArM, responded to the questions posed in the previous talks. Heterogeneity, he said, is nobody's fault - itis “just a riddle in the data that needs to be understood”. Heterogeneity testing is not new but is already requested in the E9 guidance document from the International Conference on Harmonization’, and although such testing is secondary, it requires thoughtful over the trial duration ppre-planning. He stated that, in contrast to observational meta analyses, adaptive designs have the may have had various potential to be confirmatory and hence standards for the latter need to be even higher. He felt that causes a level of significance fora heterogeneity test would be appropriate, but acknowledged that making the threshold dependent on the overall effect size is "an excellent idea” worth exploring further. However, validity and interpretation should be independent from the direction of a change. Four examples of trials ‘were given where heterogeneous resultsCoie A discussion of harmonisation acknowledged that industry and regulators have much to learn and share regarding ATDs ATDs can save six to 12 ‘months and sometimes up to two years A discussion framework wil be created bythe scientific advice working party In conclusion, Dr Koch commented that an indication of heterogeneity is not grounds for panic as it does not automatically invalidate a study. Thorough discussion between sponsor and regulator is. required to reach agreement, whether itis a chance finding or a finding that could “ill a trial”. Prof Bauer commented that in many trials with unplanned protocol amendments, heterogeneity is of greater concern than in a well-planned adaptive design. Regulators on the discussion panel supported the view of Dr Koch that, as for multiplicity, there isa price to be paid for an interim analysis with the potential to adapt the design. In reply Dr Maurer considered that, for an adaptive trial, the increased planning and logistical burden is already a price, but that the additional risk of wrongly invalidating a trial due to perceived heterogeneity should not be too high since patients could pay the price by losing @ promising new treatment. There was general agreement that heterogeneity is an important issue ‘that requires careful planning and conduct ofthe tral, but the way to investigate and weigh it, and act on it if found, needs further discussion. Harmonisation: vision for the future Christy Chuang:Stein of Pfizer led a session on “Harmonisation: Vision for the future” by acknowledging that both industry and regulators have much to learn and share regarding ATDs. It is an important desire of sponsors to conduct adaptive designs that are acceptable in multiple regions. Currently, feedback from regulators differs within the European Union and between Europe, the US and Japan, However, there are many areas where greater clarity is desirable and where regulators could hold similar views (specifically: sample size re-estimation; covariate- adaptive randomisation; adaptive dose-ranging designs; homogeneity tests; situations in which confirmatory stage-wise adaptive designs are appropriate; and sponsor involvement in data monitoring committees). By forming a worldwide expert working group to work though the issues, share examples and produce a draft global guidance document, explained Dr Chuang- Stein, it might be possible to develop this starting point into a subsequent ICH process. Such @ group should include all major stakeholders In a dynamic discussion session, Mr Hemmings reiterated that the European regulatory experience of adaptive designs, to date, has been limited and, with some exceptions, generally poorly thought through. Dr Krams acknowledged that experience is limited and that the adaptive designs working group within the US industry association PhRMA is currently collating cases to increase the experience base forall stakeholders. The role of a fully Bayesian design including the analysis received limited discussion, but it was stated that any confirmatory adaptive design must control the false positive rate and appropriate treatment effect estimates and confidence intervals need to be provided. In response to a question by Hans-Georg Eichler of the EMEA about the tangible gains from ATDs, Dr Stone replied that six to 12 months can be saved and in some cases up to one-to-two Yeats. Dr Dragalin added that adaptive dose-finding is more robust than running, a small Phase If dose- finding study and then launching into a Phase II tral, Clearly more planning time is required for adaptive designs, explained Dr Stone; that should decrease, however, as experience increases. Dr Koch agreed that the additional upfront planning and the pre-study evaluation of design operating characteristics are very important tangible benefits that adaptive designs have brought to drug. development. Closing remarks Bruno Hlamion, chair of the EMEA’s scientific advice working party and a CHMP member, provided a summary of the meeting in the form of a table (sce Table 1) capturing messages and his brief “on the spot” comments under three categories: areas of “agreement”, “disagreement”, and “open questions". Dr Flamion concluded with the information that a discussion framework will be created by the scientific advice working party, which will allow stakeholders to meet twice a year to discuss experiences and challenges relating to ATDs. This was heartily welcomed by the attendees. A suggestion from the authors is that, under this framework, the CHMP/scientific advice working party could share anonymised case examples to review good and bad practices. In conclusion, the key messages from the workshop are that ATDs are acceptable but they must be welljustiied, well-planned and well-executed, and consultation with regulators is recommended. Particular areas for care and attention include sponsor involvement in interim decision-making, and how to conduct and interpret homogeneity tests. To paraphrase Mr Hemmings, “The door is open in the right situations with the right methodology, but use ATDs carefully; good examples are still needed.” 90 Feb2000 RAJ Pharma enna worw:rajpharma.com © Informa UK Lid 2008- ice aca References 1. CHMP reflection paper on methodological sues in confirmatory clinical trials planned with an adaptive design, CHMP /EWP /2459/02 2. Maca J, Bhattacharya 5, Dragalin V, Gallo , Krams M, 2006, Adaptive seamless Phase I/II designs ~ background, operational aspects, and examples, Drug Information Journal, 0, 463-473 43. Bauer P and Kahne K, 1994, Evaluation of experiments with adaptive interim analyses, Biometrics, 50, 1029-1041 44. Brotz F Schmidt H, Kénig F, Racine A and Maures W, 2006, Confirmatory seamless phase I/II clinical ‘rials with hypotheses selection at interim: General concepts (with discussion), Biometric! Joural, 48(@), 623-634 5. Zeymer U etal, 2001, The Na+/H+ exchange inhibitor eniporide as an adjunct to exely reperfusion therapy for acute myocardial infarction, | Am Coll Crdol, 38, 1644-1660 6. Gallo F,2006, Confidentiality and Trial Integrity leues for Adaptive Designs, Drug Information Journal, 40, 445-449 7. ICHE9 Guidance, 1998, taistieal principles for clinica tral, CPMP /ICH/363/96 6‘ EGA SYMPOSIUM ON BIOSIMILAR MEDICINES 24% - 25% April 2008 sis : EVO IVa ela Ao ticiy agea ey eure rears nee TG PUPA ASA SGOT SO Rn eS RON FOR REGISTRATIONS AND FURTHER DETAILS ‘rend conjunction (© Informs UK Lis 2008 wrweraipharma.com a AN Phorma Feb 2008 94