1. Home
  2. Insights
  3. Blog
  4. Managing AI model biases in AI-enabled SaMDs

Managing AI model biases in AI-enabled SaMDs

Medical device developers using AI in software as a medical device must avoid these AI biases to optimise their SaMDs and limit health disparity perpetuation.

FDA authorisations for AI-enabled medical devices have accelerated in recent years. Technological advances make it possible for medical device developers to apply AI to enhance software as a medical device (SaMD). Used judiciously AI has many promising medical uses which can transform products, helping medics and patients to recognise symptoms, diagnose conditions and identify appropriate treatment options. Informed decision-making and strategic oversight are essential elements of developing AI in SaMD. Multiple factors, including software/hardware considerations, the device’s intended use, test model data selection and training, and many other details must be weighed long before regulatory authorisation. In this article we explore an important area of clinical consideration for AI use: AI model biases. Other topics relevant to AI in SaMD are examined in more detail in our whitepapers, links to which are included at the end of this article.

The danger of unintended AI model biases

As with any new treatments, clinicians and their patients may be wary about adopting AI-enabled SaMD. Unease about possible biases is a valid cause for concern as AI models can be biased in ways which negatively impact some patient cohorts. Medical device developers need to be aware of the possibility of these biases in AI models and should prioritise reducing and avoiding unintentional biases when working with datasets. Not all biases are undesirable; in some cases the AI algorithm can have an intentional bias designed to treat the device’s intended patient population. For example, an SaMD designed to treat pediatric patients may have an intentional pediatric patient bias. However, as noted by Cross et al., if not addressed “biased medical AI can lead to substandard clinical decisions and the perpetuation and exacerbation of longstanding healthcare disparities”.1

Imbalanced sample sizes

One of the most difficult biases to overcome is the imbalance of training, validation and testing datasets. These datasets must be representative of the patient population for the intended medical use of the device. Developers should consider carefully their device’s intended use and target population. Socioeconomic conditions will likely lead to a disproportionately low availability of data for underrepresented groups. How developers address this risk will be highly dependent on the intended use of the device. They need to weigh the risks of training a model on more data with imbalanced proportions, or less data with real world proportional datasets. The ethnic, racial, or environmental dependency of biophysical determinants (genetic and metabolic factors) that may contribute to the condition or disease of the device’s intended use will contribute to determining the best approach. Diagnosing a fractured fibula has very little (if any) dependency on the individual’s race, ethnicity, genetics or socioeconomic background whereas a Huntington’s disease diagnosis may have a high dependency on the patient’s genetic and biophysical background. In the hypothetical case of an AI-enabled SaMD for Huntington’s disease diagnosis, a demographically diverse and proportionately representative training, validation and testing dataset would be more important than for a device diagnosing fibula fractures. 

Developers need to ensure also that data is not biased in a way that could impact patients due to their gender. Some conditions present differently in women than in men and historically, women’s health has been under-researched. For example, cardiac disease has different symptoms in men and women. Therefore, both genders and the associated disease progression datasets should be represented in the AI model design and the training, validation and testing datasets. 

Misclassification of data labels

An AI algorithm can be derailed before development begins if the model datasets include inaccurate data labels. The inclusion of inaccurate labels can introduce an unintended bias or significantly reduce the model’s accuracy. To give an example, an AI model developed to diagnose multiple sclerosis (MS) that uses a training dataset that includes patients misdiagnosed with MS instead of the correct diagnosis of Lyme disease. Medical diagnoses are made by humans and therefore are prone to human error. Close attention should be paid to the selected data sets to prevent inadvertent biases associated with the inclusion of data derived from misdiagnoses. 

Historical bias

The medical field is constantly evolving based on the most up-to-date scientific evidence. According to the intended use of the SaMD being designed, developers should ensure that the medical practices have been consistent within the periods of time and regions used for the training, validation and testing datasets. A simple example would be an AI model designed to assist in the diagnosis of autism. To ensure the algorithm’s accuracy reflects our current understanding of the breadth and symptoms of autism, datasets from pre-2010 should only be considered if they have been vetted appropriately by a qualified medical professional. Historical biases are more likely with diseases and disorders at the forefront of scientific research in recent years. Including data from the 1960s for an AI algorithm intended to diagnose polio is more appropriate than using data from the same period for an AI algorithm to diagnose schizophrenia. 

Other considerations for developing AI in SaMD

These three areas of potential unintended biases are not mutually exclusive and should be considered when judging the appropriateness of dataset inclusion for AI SaMD training, validation and testing. They represent just some of the many factors that medical device developers must consider when developing their products. Other areas include selection of the most appropriate model, AI model development and testing, locked versus adaptive AI and the FDA Pre-Submission process. These are covered in depth in three ICON whitepapers: 

As medical developers enhance and optimise their products using AI and other innovative technologies, it is imperative that they do not compound existing healthcare disparities. Not only would this be a disservice to medicine, but it would squander the great potential of AI-enabled SaMD to transform healthcare.

Download our whitepaper Developing AI in SaMD to learn about other clinical considerations.

Contact us to learn more. 

  1. Cross JL, Choma MA, Onofrey JA. Bias in medical AI: Implications for clinical decision making. PLOS Digit Health. 2024;3(11):e0000651

In this section

Connect with us