“Human in the Loop” Is Not Augmented Intelligence: What It Really Takes to Bring Human Expertise to AI

By Konstantia (Nadia) Angelidou, VP Data Science, Net Health

In 2018, STAT News reported that IBM’s Watson for Oncology had produced, according to IBM’s own internal documents, “multiple examples of unsafe and incorrect treatment recommendations.” In one case, it recommended a chemotherapy drug that could cause fatal bleeding for a patient who was already bleeding. Most people working in healthcare have heard some version of this story. Fewer remember that IBM marketed Watson as augmented intelligence. What most haven’t heard is why it failed anyway.

Watson was built on human expertise. Oncologists at a leading cancer center helped train it. The problem lay in the form that expertise took. Treatment preferences from a small number of specialists, based partly on a limited set of hypothetical cases, were treated as though they represented universal clinical truth. The system was then introduced into hospitals serving different patients and working under different treatment guidelines. Its assumptions were not adequately tested before it moved into those new settings.

Watson’s failure traced back to an assumption, not an absent human in the loop: that expertise gathered in one setting would remain valid everywhere else. Human involvement is critical, but makes the biggest difference only when the right expertise enters at the right points and when assumptions continue to be tested after deployment.

The American Medical Association uses the term augmented intelligence for AI whose design “enhances human intelligence rather than replaces it.” I share the aim, but design intent is not a safeguard. Watson was built to support oncologists, not replace them, and it still failed. So here, when I use the term, I mean something narrower and harder to earn: a process in which the right human expertise enters at defined points, its assumptions are tested before deployment, and that testing continues once the model is in use. Augmentation is a property of that process, not of a person’s proximity to the model.

Whose Expertise Matters? How Is It Checked?

A data science team can help with this. They can answer whose expertise entered the model, how representative that expertise was, when it entered the process, and what happens once the world in which the model operates stops matching the world in which it was trained. A person who simply looked at the model at some point answers none of that.

That last part is the one most conversations about “human in the loop” skip. A 2026 paper in NPJ Digital Medicine on medical AI oversight makes an important distinction: a clinician reviewing an individual output can catch an error in that individual case, but that review cannot tell us whether a model is systematically wrong for a population against which it was never validated. Detecting that problem requires deliberate checks before and after deployment.

In model reviews, I often find that the hardest question is not whether the system works technically. It is whether the evidence is strong enough to justify using its output for this population, in this workflow, today. Our engineers make sure a model runs reliably, scales, and fits into the systems clinicians already use. That is a distinct and equally important discipline. The data scientist’s question runs alongside it: does the output hold up under these conditions, and how will we know when it stops holding up? A model that runs perfectly and a model that is statistically sound are two different achievements. Augmentation requires both. More than once, I have sat in a review and given an answer that was statistically right and deeply unwelcome, when a consequential product decision was on the line. Without being statistically sound, products cannot perform properly.

This gets even harder when we are not the ones who built the model. A growing share of healthcare AI relies on large language models (LLMs) developed by outside companies, using training data that customers often cannot fully inspect or verify. Standards bodies and risk-management frameworks have begun treating third-party model dependencies as a distinct source of risk. One peer-reviewed study of a widely used voice-transcription system found that roughly one in every hundred sampled transcriptions contained phrases or sentences that were not present in the original audio. Thirty-eight percent of those hallucinations included explicitly harmful content, such as perpetuating violence, creating inaccurate associations, or implying false authority.

Separate research simulated a data-poisoning attack and found that replacing just 0.001% of training tokens with medical misinformation made models more likely to propagate medical errors, while commonly used benchmarks failed to distinguish the corrupted models from their uncorrupted counterparts. These are not failures clinicians can be expected to reliably detect from a plausible-looking output alone. A fabricated sentence may read like a real one, and changes to an external model may not be fully transparent to the organization that previously validated it. Oversight must therefore begin with the evidence and change controls we require from vendors, continue through predefined validation before release, and extend to continuous monitoring of aggregate output, rather than placing our trust in any single output or review alone.

What Oversight Looks Like in Practice in Augmented Intelligence

At Net Health, we have built, and are still maturing, a governance standard for every AI feature that touches a clinical or operational decision. It is a living set of gates within the product development lifecycle that a model must clear before moving forward.

In our governance reviews, I keep returning to three questions. What evidence justifies the model’s expected performance? For whom does that evidence hold, and for whom might it not? How will we know when performance changes after release? The depth of each review scales with the feature’s risk tier. This is the general shape of the process, not the full methodology.

1. Establish the evidence before deployment: Before finalizing a model’s design, or our choice of an external model, we compare its expected performance with relevant peer-reviewed evidence and define how it will be evaluated. Before release, it undergoes statistical and clinical validation. A live client environment should not be the first place a model is asked to prove itself.
2. Test the populations and settings the model will actually serve: Bias audits and subgroup performance checks are a standard part of release, because strong average performance can conceal poor performance for a particular group. Watson’s experience illustrates the danger of carrying assumptions from one institution into very different clinical settings without sufficient validation.
3. Monitor clinical performance after release: Standard monitoring tells us whether the service is running. Clinical model monitoring answers whether the performance is changing in a way that could affect patients. That requires measures designed to detect drift and degradation, along with escalation and rollback thresholds defined before something goes wrong.

This distinction is visible in something as ordinary as a clinical note. Net Health’s ambient documentation tools listen to a therapy session and draft a structured note in real time. The draft becomes part of the record only after the clinician reviews it, corrects any errors, and signs it. Clinician review is necessary, but it is only one layer. It can catch errors in individual notes. However, it cannot tell us whether the system is becoming systematically less accurate for a particular population or setting.

That requires aggregate monitoring by the teams responsible for the model. It also requires attention to automation bias: as a tool becomes reliably useful, the second read can become less careful. Researchers have described a related “moral crumple zone” in which responsibility collapses onto the person nearest the output even when that person was not positioned to detect the underlying failure. Their sign off is a meaningful part of the process, but the review behind it must remain meaningful as well.

We’re Not Finished, and We Should Not Pretend to Be

The problem is genuinely difficult, and claims that it has been fully solved should inspire more scrutiny than confidence. What we can do is take the industry’s public failures, and the complexity beneath them, seriously enough to build testable checkpoints instead of relying on good intentions. That process is part of Net Health’s ISO/IEC 42001:2023-certified AI management system. What certification proves is a defined, auditable workflow for governing the work. It says nothing about whether the work is finished.

Why Does Process Matter to You?

Whether someone is using a risk score in a skilled nursing facility or evaluating predicted performance in a state program, they should not need to attend our validation meetings to trust the outputs present in our software. A risk score that is quietly wrong is not only a reimbursement problem. It can mean a resident who should have been flagged for a care-plan change or an intervention that should have happened earlier. Users should be able to trust that someone with the right expertise tested the output against reality before it reached them, and that someone continues to test whether it holds up as conditions change.

For “augmented intelligence” to mean anything, it must describe a process—one where human judgment remains informed, active, and accountable. It should not just imply a person’s proximity to the model. Watson had human expertise built into it. What it lacked was a process that kept testing whether the expertise still held. That is the difference between a human in the loop and augmented intelligence.

The model does not know your patients, your facility, or your state’s program. It knows the data and the assumptions on which it was built. Our responsibility is to determine where those assumptions hold, where they do not, and when changing conditions require us to intervene. My responsibility as a data scientist is to make sure that trust in that process, and in the augmented intelligence it produces, rests on evidence.