Delphi panels can be important tools for patient-centric research. In the second guidance of the United States Food and Drug Administration (FDA) Patient-Focused Drug Development (PFDD) series, Methods to Identify What is Important to Patients, the FDA briefly highlights the Delphi panel technique as another qualitative method for eliciting concepts which are important to patients.1 The Delphi panel technique can identify the most important parameters to measure in outcomes assessments and foster consensus-building through iterative rounds of surveys completed by a panel of experts.
The FDA highlights some important strengths and limitations of the Delphi panel approach, with one key limitation being the lack of universal guidelines for the process. This can make it challenging to design a Delphi study that meets the needs of all stakeholders, and in fact, there are different designs and administration methods in use across the industry. To make the most of this useful but non-standardised tool, sponsors should consider the potential Delphi panel designs and evaluate which aspects would best fit their needs.
Below, ICON’s Patient-Centred Outcomes (PCO) experts share some thoughts to assist in sponsor’s Delphi deliberations.
Classical or modified Delphi panels
The literature describes different Delphi study designs, such as the classical Delphi or modified Delphi, which may be administered by paper, email, or online,2-4 with other important considerations being the size and constitution of the expert panel, the recruitment method and approach to defining consensus. There will generally be a trade-off between cost, study timelines, and scientific rigor, so studies should be designed accordingly to meet sponsor needs.
- Classical Delphi typically uses an open-ended first round panel to elicit ideas and might consist of three or more rounds to reach consensus
- Modified Delphi typically replaces the first round with qualitative concept elicitation interviews (e.g., one-on-one, focus groups)
In the context of COA development, a modified Delphi can be advantageous in that it allows for a more thorough exploration of potentially important concepts via the qualitative interviews. This may also have the benefit of reducing the total number of rounds needed to reach consensus among the experts.
Panel administration methods
Delphi panels can be administered by paper, email, or online via an e-Delphi approach. Again, each has their advantages and limitations. For example, disseminating the Delphi survey in an online format improves the researcher’s ability to maintain respondent anonymity, which is important to reduce respondent bias. The online platform may also help streamline analysis between rounds, which can improve timelines and participant engagement. Between rounds, researchers analyse the data and update the surveys for the subsequent round—accelerating this process can keep panellists engaged and reduce attrition rates.
On the other hand, an e-Delphi approach could be costlier, as it requires additional survey programming. Also, if extensive changes to the survey are anticipated between rounds, then the additional survey programming may ultimately increase the amount of time required between rounds, even if offset by the streamlined analysis.
Constructing an expert Delphi panel
Similar to clinical trials, the inclusion criteria are key to panellist recruitment. It’s important to construct a balanced set of inclusion criteria, including the specific qualifications, years of experience in their field, number of peer-reviewed publications in their area of expertise and geographical location to find the right experts. While it’s important to ensure the participants represent true expert opinions, defining too strict inclusion criteria may impact recruitment feasibility and study timelines.
When considering the sample size for the panel, it’s also important to think about the number of anticipated rounds and plan for potential attrition of the experts. A larger sample may generally give more robust results, but it may be more challenging to achieve consensus among the larger sample.
The planned recruitment approach will also have cost and timeline impacts. If possible, recruiting experts directly through existing sponsor or CRO networks may help increase participant engagement and reduce potential non-responsiveness or attrition.
Defining consensus
Consensus is the primary objective of the Delphi process. As such, it’s important to have a clear and agreed upon definition, approach and threshold for it to know when a general agreement has been reached. It is unrealistic to expect 100% agreement among a sample, especially in contexts where standard of care may vary. A stricter threshold for consensus may necessitate the need for additional survey rounds and/or lengthen the study timelines. It is also important to consider the different types of survey questions that may be asked to drive toward consensus, such as straightforward questions with simple yes/no answers, overall relevance/importance ratings, and relative ranking exercises.
Delphi panel techniques for ClinROs
A modified Delphi may be especially useful in clinician-reported outcome (ClinRO) development. It can be used following an initial literature review and include qualitative interviews with clinical experts. This approach helps to identify the most important clinical parameters to measure and aligns with experts on how to define the levels (response options) for each parameter (item). For example, ICON PCO utilised a modified Delphi to develop a ClinRO for assessing the overall functional status of extremely premature infants while in the neonatal intensive care unit.
However, if developing a simpler ClinRO, such as a single-item disease-specific clinician global assessment of severity, a classical Delphi might be sufficient and allow for reduced study timelines. For example, ICON PCO recently utilised a similar approach to develop a single-item ClinRO for assessing the severity of metabolic crises in two related rare metabolic disorders.
Open to optimisation
The lack of universal guidelines for approaching a Delphi panel study poses a challenge but ultimately creates opportunity to design a panel that befits the needs of the specific study. Delphi techniques offer a valuable research method to efficiently gain insight into important concepts and achieve expert consensus.
ICON’s Patient-Centred Outcomes experts are well-versed in Delphi methods and can assist in optimising study design.
Connect with us to discuss Delphi designs or explore our publications below:
- Ward RM, Turner MA, Hansen-Pupp I, Higginson J, Vanya M, Flood E, Schwartz EJ, Doll HA, Tocoian A, Mangili A, Barton N, Sarda SP. Development of the PREMature Infant Index (PREMII™), a clinician-reported outcome measure assessing functional status of extremely preterm infants. J Matern Fetal Neonatal Med. 2020 Mar 5; published online.
- O'Brodovich HM, Steinhorn R, Ward RM, Hallman M, Schwartz EJ, Vanya M, et al. Development of a severity scale to assess chronic lung disease after extremely preterm birth. Pediatr Pulmonol. 2021 Mar 17. doi: 10.1002/ppul.25279.
- Schwartz EJ, Longo N, Perera S, Leuchars M, Burrow A, Ajayi T, de las Heras J, Banerjee G, Prada C, Bellenger MA, Moshkovich O, Sikirica V. Development of a Metabolic Decompensation Event Severity Score for Patients with Propionic Acidemia or Methylmalonic Acidemia. Pediatric Academic Societies Meeting 2024. E-PAS2024:622.2879.
- Longo N, Sikirica V, Grunewald S, Moshkovich O, Dionisi-Vici C, Vockley J, Bellenger MA, de las Heras J, Banerjee G, Schwartz EJ. CO67 Development of an Investigator Global Assessment of Severity in Propionic Acidemia. Value Health. 2023;26(12):S26.
References
US Food and Drug Administration (FDA). Patient-Focused Drug Development: Methods to Identify What Is Important to Patients Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders. Federal Register. 2022 Feb.
Keeney S, McKenna H, Hasson F. The Delphi Technique in Nursing and Health Research. John Wiley & Sons, 2010.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE. 2011;6(6): e20476. doi:10.1371/journal.pone.0020476
Von der Gracht HA. Consensus measurement in Delphi studies Review and implications for future quality assurance. Technological Forecasting & Social Change. 2012;79:1525-1536. doi:10.1016/j.techfore.2012.04.013
In this section
-
Digital Disruption
-
Clinical strategies to optimise SaMD for treating mental health
-
Digital Disruption: Surveying the industry's evolving landscape
- AI and clinical trials
-
Clinical trial data anonymisation and data sharing
-
Clinical Trial Tokenisation
-
Closing the evidence gap: The value of digital health technologies in supporting drug reimbursement decisions
-
Digital disruption in biopharma
-
Disruptive Innovation
- Remote Patient Monitoring
-
Personalising Digital Health
- Real World Data
-
The triad of trust: Navigating real-world healthcare data integration
-
Decoding AI in software as a medical device (SaMD)
-
Software as a medical device (SaMD)
-
Clinical strategies to optimise SaMD for treating mental health
-
Patient Centricity
-
Agile Clinical Monitoring
-
Capturing the voice of the patient in clinical trials
-
Charting the Managed Access Program Landscape
-
Developing Nurse-Centric Medical Communications
- Representation and inclusion in clinical trials
-
Exploring the patient perspective from different angles
-
Patient safety and pharmacovigilance
-
A guide to safety data migrations
-
Taking safety reporting to the next level with automation
-
Outsourced Pharmacovigilance Affiliate Solution
-
The evolution of the Pharmacovigilance System Master File: Benefits, challenges, and opportunities
-
Sponsor and CRO pharmacovigilance and safety alliances
-
Understanding the Periodic Benefit-Risk Evaluation Report
-
A guide to safety data migrations
-
Patient voice survey
-
Patient Voice Survey - Decentralised and Hybrid Trials
-
Reimagining Patient-Centricity with the Internet of Medical Things (IoMT)
-
Using longitudinal qualitative research to capture the patient voice
-
Prioritising patient-centred research for regulatory approval
-
Agile Clinical Monitoring
-
Regulatory Intelligence
-
An innovative approach to rare disease clinical development
- EU Clinical Trials Regulation
-
Using innovative tools and lean writing processes to accelerate regulatory document writing
-
Current overview of data sharing within clinical trial transparency
-
Global Agency Meetings: A collaborative approach to drug development
-
Keeping the end in mind: key considerations for creating plain language summaries
-
Navigating orphan drug development from early phase to marketing authorisation
-
Procedural and regulatory know-how for China biotechs in the EU
-
RACE for Children Act
-
Early engagement and regulatory considerations for biotech
-
Regulatory Intelligence Newsletter
-
Requirements & strategy considerations within clinical trial transparency
-
Spotlight on regulatory reforms in China
-
Demystifying EU CTR, MDR and IVDR
-
Transfer of marketing authorisation
-
Exploring FDA guidance for modern Data Monitoring Committees
-
Streamlining dossier preparation
-
An innovative approach to rare disease clinical development
-
Therapeutics insights
-
Endocrine and Metabolic Disorders
- Cardiovascular
- Cell and Gene Therapies
-
Central Nervous System
-
A mind for digital therapeutics
-
Challenges and opportunities in traumatic brain injury clinical trials
-
Challenges and opportunities in Parkinson’s Disease clinical trials
-
Early, precise and efficient; the methods and technologies advancing Alzheimer’s and Parkinson’s R&D
-
Key Considerations in Chronic Pain Clinical Trials
-
ICON survey report: CNS therapeutic development
-
A mind for digital therapeutics
-
Glycomics
- Infectious Diseases
- NASH
- Obesity
- Oncology
- Paediatrics
-
Respiratory
-
Rare and orphan diseases
-
Advanced therapies for rare diseases
-
Cross-border enrollment of rare disease patients
-
Crossing the finish line: Why effective participation support strategy is critical to trial efficiency and success in rare diseases
-
Diversity, equity and inclusion in rare disease clinical trials
-
Identify and mitigate risks to rare disease clinical programmes
-
Leveraging historical data for use in rare disease trials
-
Natural history studies to improve drug development in rare diseases
-
Patient Centricity in Orphan Drug Development
-
The key to remarkable rare disease registries
-
Therapeutic spotlight: Precision medicine considerations in rare diseases
-
Advanced therapies for rare diseases
-
Endocrine and Metabolic Disorders
-
Transforming Trials
-
Accelerating biotech innovation from discovery to commercialisation
-
Ensuring the validity of clinical outcomes assessment (COA) data: The value of rater training
-
Linguistic validation of Clinical Outcomes Assessments
-
Optimising biotech funding
- Adaptive clinical trials
-
Best practices to increase engagement with medical and scientific poster content
-
Decentralised clinical trials
-
Biopharma perspective: the promise of decentralised models and diversity in clinical trials
-
Decentralised and Hybrid clinical trials
-
Practical considerations in transitioning to hybrid or decentralised clinical trials
-
Navigating the regulatory labyrinth of technology in decentralised clinical trials
-
Biopharma perspective: the promise of decentralised models and diversity in clinical trials
-
eCOA implementation
-
Blended solutions insights
-
Clinical trials in Japan: An enterprise growth and management strategy
-
How investments in supply of CRAs is better than competing with the demand for CRAs
-
The evolution of FSP: not just for large pharma
-
Embracing a blended operating model
-
Observations in outsourcing: Survey results show a blended future
-
Clinical trials in Japan: An enterprise growth and management strategy
-
Implications of COVID-19 on statistical design and analyses of clinical studies
-
Improving pharma R&D efficiency
-
Increasing Complexity and Declining ROI in Drug Development
-
Innovation in Clinical Trial Methodologies
- Partnership insights
-
Risk Based Quality Management
-
Transforming the R&D Model to Sustain Growth
-
Behind Biotech: Stories of science and resilience
-
Accelerating biotech innovation from discovery to commercialisation
-
Value Based Healthcare
-
Strategies for commercialising oncology treatments for young adults
-
US payers and PROs
-
Accelerated early clinical manufacturing
-
Cardiovascular Medical Devices
-
CMS Part D Price Negotiations: Is your drug on the list?
-
COVID-19 navigating global market access
-
Ensuring scientific rigor in external control arms
-
Evidence Synthesis: A solution to sparse evidence, heterogeneous studies, and disconnected networks
-
Health technology assessment
-
Perspectives from US payers
-
ICER’s impact on payer decision making
-
Making Sense of the Biosimilars Market
-
Medical communications in early phase product development
-
Navigating the Challenges and Opportunities of Value Based Healthcare
-
Payer Reliance on ICER and Perceptions on Value Based Pricing
-
Payers Perspectives on Digital Therapeutics
-
Precision Medicine
-
RWE Generation Cross Sectional Studies and Medical Chart Review
-
Survey results: How to engage healthcare decision-makers
-
The affordability hurdle for gene therapies
-
The Role of ICER as an HTA Organisation
-
Integrating openness and precision for competitive advantage
-
Strategies for commercialising oncology treatments for young adults
-
Blog
-
Videos
-
Webinar Channel