How Are Healthcare Leaders Tackling Automation Bias?

Healthcare organizations are using AI more than ever before, but plenty of questions remain when it comes to ensuring the safe, responsible use of these models. Industry leaders are still working to figure out how to best address concerns about algorithmic bias, as well as liability if an AI recommendation ends up being wrong.

During a panel discussion last month at MedCity News’ INVEST Digital Health conference in Dallas, healthcare leaders discussed how they are approaching governance frameworks to mitigate bias and unintended harm. They think that the key pieces are vendor responsibility, better regulatory compliance and clinician engagement.

Ruben Amarasingham — CEO of Pieces Technologies a healthcare AI startup acquired by Smarter Technologies last week — noted that while human-in-the-loop systems can help curb bias in AI, one of the most insidious risks is automation bias, which refers to people’s tendency to overtrust machine-generated recommendations. 

“One of the biggest examples in the commercial consumer industry is GPS maps. Once those were introduced, when you study cognitive performance, people would lose spatial knowledge and spatial memory in cities that they’re not familiar with — just by relying on GPS systems. And we’re starting to see some of those things with AI in healthcare,” Amarasingham explained.

Automation bias can lead to “de-skilling,” or the gradual erosion of clinicians’ human expertise, he added. He pointed to research from Poland that was published in August showing that gastroenterologists using AI tools became less skilled at identifying polyps.

Amarasingham believes that vendors have a responsibility to monitor for automation bias by analyzing their users’ behavior.

“One of the things that we’re doing with our clients is to look at the acceptance rate of the recommendations. Are there patterns that suggest that there’s not really any thought going into the acceptance of the AI recommendation? Even though we might want to see a 100% acceptance rate, that’s probably not ideal — that suggests that there isn’t the quality of thought there,” he declared.

Alya Sulaiman, chief compliance and privacy officer at health data platform Datavant, agreed with Amarasingham, saying that there are legitimate reasons to be concerned that healthcare personnel could blindly trust AI recommendations or use systems that effectively operate on autopilot. She noted that this has led to numerous state laws imposing regulatory and governance requirements for AI, including notice, consent and strong risk assessment programs.

Sulaiman recommended that healthcare organizations clearly define what success looks like for an AI tool, how it could fail, and who could be harmed — which can be a deceptively difficult task because stakeholders often have different perspectives.

“One thing that I think we will continue to see as both the federal and the state landscape evolves on this front, is a shift towards use case-specific regulation and rulemaking — because there’s a general recognition that a one-size-fits-all approach is not going to work,” she stated.

For instance, we might be better off if mental health chatbots, utilization management tools and clinical decision support models all had their own set of unique government principles, Sulaiman explained.

She also highlighted that even administrative AI tools can create harm if errors occur. For example, if an AI system misrouted medical records, it could send a patient’s sensitive information to the wrong recipient, and if an AI model incorrectly processed a patient’s insurance data, it could lead to delays in care or billing mistakes.

While clinical AI use cases often get the most attention, Sulaiman stressed that healthcare organizations should also develop governance frameworks for administrative AI tools — which are rapidly evolving in a regulatory vacuum. 

Beyond regulatory and vendor responsibilities, human factors — like education, trust building and collaborative governance — are critical to ensuring AI is deployed responsibly, said Theresa McDonnell, Duke University Health System’s chief nurse executive.

“The way we tend to bring patients and staff along is through education and being transparent. If people have questions, if they’ve got concerns, it takes time. You have to pause. You have to make sure that people are really well informed, and at a time when we’re going so fast, that puts additional stressors and burdens on the system — but it’s time well worth taking,” McDonnell remarked.

All panelists agreed that oversight, transparency and engagement are crucial to safe AI adoption.

Photo: MedCity News