AI’s Healthcare ‘Doomsday’? It May Be Less About Autonomy Than Trust

The recent warnings about “AI doomsday” may sound distant from the messy realities of a busy hospital or clinic, but as artificial intelligence moves deeper into clinical care, the question of how much control humans should retain and how AI is implemented into healthcare workflows is quickly becoming an urgent issue. 

AI is already being used to help clinicians process information, manage administrative tasks, and capture patterns in patient data. As those capabilities expand into the more complex tasks of diagnosis and treatment, the stakes get higher. Up until recently, Dr. David Kirk, an ICU physician for more than 20 years, used to be an AI skeptic. As chief medical officer at NYC-based company Regard, Kirk now views it as an essential tool for helping physicians manage the enormous amount of information involved in day-to-day patient care. Here’s why his thinking has changed. 

“Having an additional set of eyes that can sift through huge amounts of information and identify patterns that might otherwise be missed is so powerful that it is unethical not to use it at this point.”, Kirk said. 

That doesn’t mean handing over responsibility for clinical decisions to an AI system. Instead, Kirk sees its value in combining AI’s efficient processing with the physician’s real-world experience and knowledge of the patient as an individual. 

As healthcare organizations deploy increasingly autonomous systems, a common talking point is the idea of ‘human-in-the-loop.’ That’s the concept that a human eye should always review and evaluate the outputs of any healthcare AI system. Kirk argues that the phrase can oversimplify how clinical oversight actually works. 

“The degree of autonomy should be directly proportional to the degree of risk,” he said. 

Healthcare already operates this way. Nurses, pharmacists, and other members of the care team are both empowered to make certain decisions independently, and also obligated to address concerns when something isn’t right rather than blindly following a physician’s orders. Kirk asserts that AI should integrate to this existing culture of team-based oversight.  

“As AI advances in assisting healthcare, this ability to raise a warning by any member of the team must be protected and encouraged.” Kirk said. 

For Kirk, the more immediate danger isn’t an autonomous system taking over medicine and leaving human decision-making behind. It’s healthcare organizations harnessing AI primarily to increase efficiency while leaving healthcare providers even more mired in complexity and regulation, effectively adding to their workload rather than lightening it. 

“There is an incredibly high risk that AI will be mainly used to maximize efficiency, increase regulations, and magnify complexity. Healthcare will always require humans caring for humans, but today, those in healthcare are too buried in clerical work for that to happen comfortably,” he said. “AI could dramatically restore physicians’ and patients’ ability to develop a trusting relationship, but we must make that a priority.” 

With proper guardrails and team-based oversight in place, Kirk asserts that AI could give physicians more time to do the work that cannot be automated: connecting on a human level with patients, exercising judgment rooted in experience, and building trust. 

As the biggest leaders in AI warn of its potential threat to humanity, healthcare offers a clear example of how it can instead support human interactions. The closer AI gets to clinical decision-making, the more care must be used in implementing safe limits and ensuring that it supports rather than impedes the human side of medicine. 

Photo: ismagilov, Getty Images