AI Workers Need a Probation Period, Not Just a Pilot

Recently, I was walking the leadership team at a clinic through the rollout plan for a new AI Admissions Coordinator. I went through the sequence almost mechanically: first a pilot, then probation, then full deployment. The CEO stopped me. “What do you mean by probation?”

The question caught me off guard. After spending the past year deploying AI workers in mental health clinics, the distinction seemed clear to me. It wasn’t. Healthcare leaders know how to test software. We don’t yet have a shared playbook for onboarding agents expected to do a job end to end. That is the gap a probation period is meant to fill.

Healthcare is already good at running pilots and much less consistent at turning them into daily operations. A survey of more than 400 U.S. healthcare leaders found that only 30% of completed AI proofs of concept had made it into production. The reasons ranged from security and data readiness to integration costs and limited in-house expertise. A probation period won’t fix all of them. It addresses a narrower problem: a team proves that the technology can handle the expected workflow and assumes the implementation is done. Once the demo works, the pressure comes off. In reality, the work of monitoring real cases, correcting mistakes ​​has only just begun.

A recent MedCity article made a strong case that workflow integration is where many AI deployments break down. I agree. But with agents, integration only gets you to the starting line. Giving a system access to the right tools does not mean it is ready to use them without routine intervention.

Take an AI Admissions Coordinator as an example. Its job is not simply to answer the phone. It has to know when it has enough information to move forward, route a patient correctly, book the right appointment, and recognize when a case falls outside the rules it has been given. The happy path is usually straightforward. The harder questions show up when insurance information is incomplete, a request does not fit neatly into the clinic’s scheduling rules, or the next step is unclear. That is where a pilot ends and probation begins.

Probation is the period when the clinic helps the agent succeed in those real situations. Start small: one location, a defined set of call types, or a limited group of appointment categories. At first, the team may review every call. That is not a sign that the pilot failed; it’s how onboarding works. What the team learns should feed back into the agent’s instructions, integrations, and escalation rules. As performance becomes consistent, the clinic can expand the volume and variety of work while stepping back from routine review. Eventually, routine calls should no longer need intervention. The team shifts from observing calls to monitoring overall metrics, while the agent flags the exceptions.

The rollout also changes who should sign off. The person who approves the pilot is not necessarily the person who ends probation. A CEO or COO can assess the business case, strategic fit, and risk tolerance. The workflow owner is better placed to judge the work itself. For an Admissions Coordinator, that may be the VP or Head of Admissions – the person who sees incomplete cases, downstream cleanup, and avoidable escalations. Senior leadership decides whether the organization wants the agent. The workflow owner decides whether the team can trust it with the job every day.

The scorecard is another place where old software habits get in the way. With conventional software, teams often look at logins, daily active users, feature adoption, and uptime. Those measures are useful, but they do not show whether an agent is finishing the work. During probation, one needs to track the share of eligible cases completed end to end, the accuracy and completeness of those cases, downstream rework, whether the right exceptions were escalated, and the human time required per completed case.

Those numbers show how much dependable capacity the agent is actually adding to the team. That capacity might equal the output of one full-time employee or five. But the comparison only holds if the work meets the same quality. Calling something “five FTEs of capacity” while employees quietly clean up its work is accounting fiction.

Graduating from probation does not end oversight. The Joint Commission’s Responsible Use of AI certification includes ongoing monitoring of AI performance and safety across its lifecycle. Clinics should run their operations with the same mindset. When an agent takes on a new workflow, treat it like a promotion and put that new responsibility through probation. Do the same after a material change to the model, instructions, integrations, or the clinic’s own rules. Past performance tells you that the previous setup worked. It does not automatically validate the new one.

That does not mean going back to reviewing every case forever. Mature work can stay exception-based. New responsibilities or meaningful changes should temporarily bring back closer supervision, and the workflow owner should again decide when the agent has earned more autonomy.

If that CEO asked me the same question about “probation” today, I would use the hiring analogy. A pilot is the CV, interview, and work sample. You learn whether the agent has the necessary skills and can handle the expected task under controlled conditions. Probation is the first stretch on the job. That is when you find out whether it can handle normal variation, finish the work reliably, and ask for help at the right time. The pilot gives you evidence of capability. Probation tells you whether the agent is ready for responsibility.

Photo: Sandwish, Getty Images

This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.