The Big Challenge for AI in Healthcare Comes After Approval

For years, much of the work around enabling AI in healthcare has focused on getting approval despite the sparse regulations and guidelines available. Is the ML module part of the medical device? What evidence will we need to show to regulators? Which regulatory pathway is best for this? With the FDA having authorized roughly 1,450 AI enabled devices to date, ~295 in 2025 alone vs. ~6 in 2015, our focus now needs to shift towards what happens to these devices after they’ve been released for clinical use.

Typically, software placed on the market will undergo changes and updates after its initial release. In the case of healthcare AI, changes in performance can occur, often unexpectedly, even when the developers have not updated the model. Because a healthcare AI system is intended for use in clinical settings, it will be exposed to a wide variety of patients and clinical situations (dataset shift). Even a well-designed validation cannot fully anticipate every patient population, site, or clinical practice the system will face after deployment — so it will typically be operating in conditions that differ from the one used for the model’s validation. The question then is: what happens to the performance of a healthcare AI system when it is operating in a real world environment as opposed to the controlled environment used for the model’s validation?

Some interesting numbers came out of a recent study of an authorized autonomous system for the detection of diabetic retinopathy from retinal images. The system failed to produce results for a quarter of the patients in the study. Surprisingly, this was not because the underlying model had not been retrained or updated for several years and therefore did not perform well on new cases. Rather, the camera quality and the size of the patient’s pupil meant that proper imaging was not possible in a quarter of the cases. So, the underlying model performed well on new cases, but because of differences between the environment in which it was used and the environment in which it was validated, its performance in practice was affected. This is, essentially, a change control issue for AI in healthcare.

AI doesn’t change the basic questions around change control 

The core questions that need to be addressed when medical device’s software changes — including intended purpose, clinical functionality and risk — remain the same when AI enabled systems are involved. Assessing the impact of changes to an AI model or it’s training dataset is, however, more difficult compared to rule-based algorithm. That difficulty stems largely from how tightly the two are entangled with one another.

The model and its dataset exist in a continuous feedback loop, each pushing the other to change. A shift in real-world data can cause the model’s performance to drift, which triggers retraining on updated data — and that retraining produces a model with new parameters (different weights), which must again be validated against the very data that set the cycle in motion. Rarely does one change without the other following

I mentioned earlier that an update to an AI model supporting clinical function does not necessarily need to be released as a new medical device. In fact, manufacturers should pre-define the framing of possible changes prior to release. Later, an assessment of the changes made to the model in the context of what the updated model will do is required. This assessment will help determine whether further validation is required. As an example, if the updated AI model is used for diagnosis, then an update that may seem trivial to implement by the engineering team could have significant implications for the risk profile or clinical use of the medical device and therefore require further validation.

It is much better to have these discussions when designing an update to the AI model than when the software update is about to be released and a large team of people is required to gather evidence in a very short space of time to prove that the appropriate amount of validation has taken place. Least to say, such approach may generate significant errors and nonconformities and is contradictory to general best practices.

Regulators are thinking about change too 

The FDA’s framework for a Predetermined Change Control Plan for certain software functions of AI enabled medical devices enables manufacturers to plan for expected modifications to authorized software functions in advance of their implementation. Such planned modifications can be described in the marketing submission and outline the methods for the development and validation of changes to the authorized functions. It is, however, expected that the manufacturer will continue to monitor the modified functions to verify that they continue to perform within the intended parameters for which they were reviewed and authorized.

In the EU, updates to AI enabled medical software need to be assessed for compliance with the requirements of MDR and IVDR and, in due course, with the applicable requirements of the EU AI Act. Therefore, it is important to consider the evolution of an AI system already at the design stage in order to anticipate potential problems when updates to the system are implemented.

But this is also relevant beyond the US and EU. The International Medical Device Regulators Forum (IMDRF) has developed guidance on Predetermined Change Control Plans for machine learning enabled medical devices that can be applied across markets. This can be particularly useful for manufacturers operating in multiple countries, including markets where AI specific regulation is still developing or where existing rules provide limited detail on post market changes.

IMDRF’s guidance also makes an important connection between change control and the data behind the model. Its work on machine learning enabled medical devices, together with the joint guiding principles on Predetermined Change Control Plans published by the FDA, Health Canada and the UK’s MHRA, considers how training, validation and test datasets are managed over time. For manufacturers, this means a change control framework cannot focus only on the model while treating the data behind it as a separate issue. The two need to be considered together when assessing how a system may change after release.

That said, generative AI, in particular, has not yet received authorization for any medical devices in the US. Generative AI is starting to enter the regulatory discourse, with the first FDA Breakthrough Device designation for a patient-facing clinical generative AI application announced in March 2026. Even at this early stage, therefore, it is becoming increasingly clear that questions around how to manage these sorts of systems over time are not going to go away.

Change control belongs in the AI strategy 

Organizations with established quality systems for managing regulated healthcare products will find a solid foundation to build on in this area. However, the important question remains whether these processes can also account for changes in performance that AI systems can develop without any intervention from developers, even when no changes have been made to the model.

I also think it is essential for regulatory and quality specialists to get involved earlier in the development or re-validation of AI healthcare products. By understanding how a system is intended to evolve during its development, the necessary steps to validate future changes can be outlined and the appropriate time for further regulatory assessment can be established.

Authorization of an AI-enabled healthcare product is not a point in time event and will require the manufacturer to monitor and manage the product after its release for clinical use. For healthcare AI, understanding when sufficient change has occurred in the product’s performance or other characteristics may ultimately require as much discipline and resources as gaining approval in the first place.

Photo: marchmeena29, Getty Images

This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.