Designing Delphi panels

Delphi panels can be important tools for patient-centric research. In the second guidance of the United States Food and Drug Administration (FDA) Patient-Focused Drug Development (PFDD) series, Methods to Identify What is Important to Patients, the FDA briefly highlights the Delphi panel technique as another qualitative method for eliciting concepts which are important to patients.¹ The Delphi panel technique can identify the most important parameters to measure in outcomes assessments and foster consensus-building through iterative rounds of surveys completed by a panel of experts.

The FDA highlights some important strengths and limitations of the Delphi panel approach, with one key limitation being the lack of universal guidelines for the process. This can make it challenging to design a Delphi study that meets the needs of all stakeholders, and in fact, there are different designs and administration methods in use across the industry. To make the most of this useful but non-standardised tool, sponsors should consider the potential Delphi panel designs and evaluate which aspects would best fit their needs.

Below, ICON’s Patient-Centred Outcomes (PCO) experts share some thoughts to assist in sponsor’s Delphi deliberations.

Classical or modified Delphi panels

The literature describes different Delphi study designs, such as the classical Delphi or modified Delphi, which may be administered by paper, email, or online,^2-4 with other important considerations being the size and constitution of the expert panel, the recruitment method and approach to defining consensus. There will generally be a trade-off between cost, study timelines, and scientific rigor, so studies should be designed accordingly to meet sponsor needs.

Classical Delphi typically uses an open-ended first round panel to elicit ideas and might consist of three or more rounds to reach consensus
Modified Delphi typically replaces the first round with qualitative concept elicitation interviews (e.g., one-on-one, focus groups)

In the context of COA development, a modified Delphi can be advantageous in that it allows for a more thorough exploration of potentially important concepts via the qualitative interviews. This may also have the benefit of reducing the total number of rounds needed to reach consensus among the experts.

Panel administration methods

Delphi panels can be administered by paper, email, or online via an e-Delphi approach. Again, each has their advantages and limitations. For example, disseminating the Delphi survey in an online format improves the researcher’s ability to maintain respondent anonymity, which is important to reduce respondent bias. The online platform may also help streamline analysis between rounds, which can improve timelines and participant engagement. Between rounds, researchers analyse the data and update the surveys for the subsequent round—accelerating this process can keep panellists engaged and reduce attrition rates.

On the other hand, an e-Delphi approach could be costlier, as it requires additional survey programming. Also, if extensive changes to the survey are anticipated between rounds, then the additional survey programming may ultimately increase the amount of time required between rounds, even if offset by the streamlined analysis.

Constructing an expert Delphi panel

Similar to clinical trials, the inclusion criteria are key to panellist recruitment. It’s important to construct a balanced set of inclusion criteria, including the specific qualifications, years of experience in their field, number of peer-reviewed publications in their area of expertise and geographical location to find the right experts. While it’s important to ensure the participants represent true expert opinions, defining too strict inclusion criteria may impact recruitment feasibility and study timelines.

When considering the sample size for the panel, it’s also important to think about the number of anticipated rounds and plan for potential attrition of the experts. A larger sample may generally give more robust results, but it may be more challenging to achieve consensus among the larger sample.

The planned recruitment approach will also have cost and timeline impacts. If possible, recruiting experts directly through existing sponsor or CRO networks may help increase participant engagement and reduce potential non-responsiveness or attrition.

Defining consensus

Consensus is the primary objective of the Delphi process. As such, it’s important to have a clear and agreed upon definition, approach and threshold for it to know when a general agreement has been reached. It is unrealistic to expect 100% agreement among a sample, especially in contexts where standard of care may vary. A stricter threshold for consensus may necessitate the need for additional survey rounds and/or lengthen the study timelines. It is also important to consider the different types of survey questions that may be asked to drive toward consensus, such as straightforward questions with simple yes/no answers, overall relevance/importance ratings, and relative ranking exercises.

Delphi panel techniques for ClinROs

A modified Delphi may be especially useful in clinician-reported outcome (ClinRO) development. It can be used following an initial literature review and include qualitative interviews with clinical experts. This approach helps to identify the most important clinical parameters to measure and aligns with experts on how to define the levels (response options) for each parameter (item). For example, ICON PCO utilised a modified Delphi to develop a ClinRO for assessing the overall functional status of extremely premature infants while in the neonatal intensive care unit.

However, if developing a simpler ClinRO, such as a single-item disease-specific clinician global assessment of severity, a classical Delphi might be sufficient and allow for reduced study timelines. For example, ICON PCO recently utilised a similar approach to develop a single-item ClinRO for assessing the severity of metabolic crises in two related rare metabolic disorders.

Open to optimisation

The lack of universal guidelines for approaching a Delphi panel study poses a challenge but ultimately creates opportunity to design a panel that befits the needs of the specific study. Delphi techniques offer a valuable research method to efficiently gain insight into important concepts and achieve expert consensus.

ICON’s Patient-Centred Outcomes experts are well-versed in Delphi methods and can assist in optimising study design.

Connect with us to discuss Delphi designs or explore our publications below:

Ward RM, Turner MA, Hansen-Pupp I, Higginson J, Vanya M, Flood E, Schwartz EJ, Doll HA, Tocoian A, Mangili A, Barton N, Sarda SP. Development of the PREMature Infant Index (PREMII™), a clinician-reported outcome measure assessing functional status of extremely preterm infants. J Matern Fetal Neonatal Med. 2020 Mar 5; published online.
O'Brodovich HM, Steinhorn R, Ward RM, Hallman M, Schwartz EJ, Vanya M, et al. Development of a severity scale to assess chronic lung disease after extremely preterm birth. Pediatr Pulmonol. 2021 Mar 17. doi: 10.1002/ppul.25279.
Schwartz EJ, Longo N, Perera S, Leuchars M, Burrow A, Ajayi T, de las Heras J, Banerjee G, Prada C, Bellenger MA, Moshkovich O, Sikirica V. Development of a Metabolic Decompensation Event Severity Score for Patients with Propionic Acidemia or Methylmalonic Acidemia. Pediatric Academic Societies Meeting 2024. E-PAS2024:622.2879.
Longo N, Sikirica V, Grunewald S, Moshkovich O, Dionisi-Vici C, Vockley J, Bellenger MA, de las Heras J, Banerjee G, Schwartz EJ. CO67 Development of an Investigator Global Assessment of Severity in Propionic Acidemia. Value Health. 2023;26(12):S26.

References

US Food and Drug Administration (FDA). Patient-Focused Drug Development: Methods to Identify What Is Important to Patients Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders. Federal Register. 2022 Feb.
Keeney S, McKenna H, Hasson F. The Delphi Technique in Nursing and Health Research. John Wiley & Sons, 2010.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS ONE. 2011;6(6): e20476. doi:10.1371/journal.pone.0020476
Von der Gracht HA. Consensus measurement in Delphi studies Review and implications for future quality assurance. Technological Forecasting & Social Change. 2012;79:1525-1536. doi:10.1016/j.techfore.2012.04.013

Designing Delphi panels: A useful, non-standardised tool for patient-centric research