Human-in-the-Loop, But Not in Control

TL;DR

Human-in-the-Loop (HITL) is frequently presented as a safeguard for AI and other automated systems. But simply inserting a person into an approval process doesn’t necessarily make the system safer, more accurate, or more accountable. Sometimes it’s little more than a checkbox.

Effective HITL requires more than a qualified person with the authority to intervene. That person must also have sufficient information, situational awareness, time, and practical ability to exercise meaningful judgment. Here’s the paradox: the more successfully we automate a task, the less capable its human supervisor may become of intervening when something goes wrong. This isn’t a new problem. Human-factors researchers were warning about it decades before modern AI. What’s changing is the speed, scale, complexity, and increasing autonomy of the systems we’re deploying.

If we’re going to insist on keeping humans in the loop, we need to design systems that keep them capable of being there. We’ve already seen real-world automation failures in systems that were supposed to have human oversight. In some cases, that oversight proved ineffective when it was needed most.

That’s the premise. If you want the deeper version, read on…

The Human Checkpoint

Sometimes HITL may be a load of garbage. It’s technically there, but the human checkpoint is little more than a formality. Maybe it’s just a corporate checkbox. A procedural step that allows an organization to claim there’s a human checkpoint somewhere in the process. But does that person add value? Do they improve outcomes? Are they providing the safety gate we think we’re getting? Or are we just inserting an approval button between an automated recommendation and an automated action?

The term Human-in-the-Loop is used somewhat loosely. Sometimes it’s a person actively collaborating with a system. Other times it’s someone reviewing a recommendation, supervising a process, or simply acknowledging a result before it moves forward.

Those are very different responsibilities.

I think meaningful human oversight comes down to three fundamental dimensions:

Awareness: Can the person recognize when something is wrong? Do they have the information and situational awareness necessary to understand what’s happening?
Capability: Do they have the training, experience, and retained skills to determine what should be done about it?
Control: Do they have the authority, tools, and practical ability to intervene effectively and in time?

These aren’t independent checkboxes. Someone might recognize a problem but lack expertise to resolve it. They might know exactly what to do but lack authority or tools to act. Or they might have complete control over a system without sufficient awareness to recognize when intervention is necessary. There’s also a more troubling possibility. Someone may recognize a problem, understand how to intervene, and still hesitate because challenging the system carries personal or professional consequences. We need these capabilities, and also ensure these capabilities don’t disappear as automation becomes more effective.

All three must exist at the moment meaningful intervention is required. (For real, not only as a thin attempt at a liability shield.)

When Is HITL Important?

Not every automated decision needs a person supervising it. And not every human checkpoint improves results. We automate for good reasons. Automation can perform repetitive tasks more quickly, consistently, and accurately than people. It can handle the traditional three D’s of automation: work that is Dull, Dirty, or Dangerous.

Reintroducing human effort into every automated step would defeat much of the purpose. A better question is where human judgment adds something automation doesn’t reliably provide.

That may include:

High-consequence decisions. Actions involving physical safety, substantial financial exposure, legal rights, medical treatment, or other significant consequences.
Ambiguous situations. Cases where context, conflicting goals, or unusual circumstances make a confident automated recommendation difficult to trust.
Irreversible or expensive actions. Decisions that cannot easily be undone, or where recovery would be particularly costly.
Novel situations. Circumstances outside the system’s expected operating conditions or where historical examples offer limited guidance.
Decisions requiring accountable human judgment. Situations where responsibility, discretion, or legal requirements justify human participation.

Categories overlap. None automatically establishes that HITL is the best approach. A system that executes an incorrect decision 10,000 times before requesting human approval hasn’t necessarily become safer by adding a human at the end. Conversely, a well-constrained automated process with independent validation, limited permissions, and reliable exception handling may be safer than one requiring a person to approve every ordinary action.

The goal should be to introduce meaningful human judgment where it improves the system, while using other controls where they work better.

Lessons From Failures

We’ve already seen real-world automation failures in systems supposedly protected by human oversight. And before you say I’m playing FUD games with rare exceptions, these aren’t isolated to a single industry or technology, and some have been enormously costly. I just don’t want you or me to be the next front-page example.

In 2018, an Uber autonomous test vehicle struck and killed a pedestrian despite having a human safety operator behind the wheel. The NTSB cited operator distraction, inadequate supervision, and failure to address automation complacency. In 2012, Knight Capital lost more than $460 million in roughly 45 minutes when automated trading software malfunctioned. The SEC investigation revealed serious deficiencies in deployment procedures, monitoring, and risk controls.

Now let’s look at more recent examples.

In 2025, Deloitte delivered an AI-assisted report to the Australian government containing fabricated academic references and inaccurate legal material. The errors got past any professional review, were identified by an outside researcher, and resulted in a partial refund. That same year, Replit’s AI coding agent deleted a production database containing records for more than 1,200 executives, despite explicit human instructions prohibiting changes. The database was eventually restored, but the incident showed how human instructions can be ineffective without enforceable technical controls.

Want more recent? As I write this, just last week, October 2, 2026, federal judge Arun Subramanian warned that reliance on AI could undermine junior lawyers’ professional development, potentially weakening the expertise needed for future oversight.

These failures aren’t identical. Some involve inadequate review, others insufficient technical controls. And concerns about skill degradation remain a longer-term risk rather than an established cause of these incidents. But the lesson is consistent: human involvement, responsibility, and meaningful control are three different things.

The Automation Paradox: It’s Not a New Thing

In 1983, researcher Lisanne Bainbridge published Ironies of Automation. This was long before ChatGPT, large language models, or today’s autonomous agent spread. Bainbridge was examining industrial automation and the changing role of human operators. One of her central observations was removing people from routine operation could leave them responsible for precisely the situations they were becoming least prepared to handle. (If you’ve followed any of my writing, you know I’m a big fan of irony. I’m not so sure this time though.)

Think about what happens when a system gradually takes over more work. The automation becomes responsible for normal operations. People stop performing the routine tasks. Instead, they monitor, occasionally make decisions, and intervene when exceptions occur. There are obvious productivity benefits. But something else can happen. People lose the regular practice that helped them develop and maintain expertise. Their understanding of the system may become less current. Their situational awareness can deteriorate. And when something finally goes wrong, they’re expected to take over immediately. That may not be a reasonable expectation unless there’s mitigating actions, such as recurrent training.

The human’s job shifts from doing the work to rescuing the automation when the work becomes difficult.

This is often described as the automation paradox, or more specifically, one of the ironies of automation. And it isn’t just a theoretical concern. In 1995, Mica Endsley and Esin Kiris published The Out-of-the-Loop Performance Problem and Level of Control in Automation. Their research examined how automation could reduce an operator’s ability to respond after a system failure. Among their findings was the importance of situational awareness and the problems associated with shifting from active participation to passive monitoring.

So the fundamental challenge predates modern AI by decades. What interests me is whether today’s AI systems are accelerating this problem, and whether increasingly complex layers of automation are creating additional ways for humans to lose control without realizing it.

Maintaining “the Flick”

There’s an expression in air traffic control: having the flick, sometimes described as having the picture. It refers to a controller’s mental understanding of what’s happening in their assigned airspace. This isn’t just where aircraft are at this instant, but where they’re going, how they’re moving relative to one another, what clearances they’ve received, and what conflicts may be developing.

It’s a dynamic mental model of three-dimensional space and time. A controller can have every aircraft displayed on a radar screen and still lose the flick. They can see the individual pieces without maintaining the overall picture. Workload, fatigue, distractions, and unexpected events can all contribute to this loss of awareness. Research documented through NASA’s Behavioral Indicators: How You Know When You Are Losing the Flick and What to Do About It? examines indicators of performance decline and the influences affecting controllers.

I think the analogy to AI oversight is useful. Consider someone supervising an AI agent that handles customer refunds. The agent reviews purchase history, checks eligibility, searches internal policies, evaluates possible fraud indicators, and recommends approval.

The human sees: Refund requested: $875. Recommended action: Approve.

There’s an approval button and a rejection button. The person has technical authority to intervene. They may even understand the refund policy perfectly. But do they know which records the agent examined? Did the agent correctly identify the customer? Were there conflicting transactions? Did it apply the current policy? Has another agent already issued a partial refund? They have an answer. They may not have the picture. And an especially important distinction emerges here: being able to see the result of an automated process is not the same as understanding enough of the process to supervise it.

That doesn’t mean humans must watch every calculation or every software operation. Usually they shouldn’t. And the reality is that increasingly, we can’t. It means the interface and workflow must give the context necessary to recognize the kinds of errors they’re supposed to catch. Otherwise, we’re asking a person to exercise judgment without giving them what that judgment requires.

Don’t Be an Approval Monkey

One danger of HITL is that it gradually turns people into approval monkeys.

Approve. Approve. Approve. Come on now, we’ve all likely done this. Right now, as I edit this article I’m running multiple automations in another window. I can see some scrolling out the of the corner of my eye. When it stops and the little status light goes green, I hop over there. Often, just to take a quick look and click, “Continue.” At certain stage checks I get more of a report and maybe have to make some real choices. (Maybe I should set it up to pop out a piece of chocolate or something whenever I click. Or perhaps a banana.) Meanwhile, I’m maybe annoyed because I was in the middle of this paragraph. Should I finish it first? Probably, but writing is just for fun and I need to get the other thing done. Did I give it all the attention it deserved? I like to think so. Or is that my own confirmation bias?

Most of the recommendations are fine. The system is generally reliable. The reviewer gets used to accepting what it produces. The thing is, (and we know this), human attention isn’t unlimited. And we can be easily distracted.

Imagine reviewing hundreds of AI-generated recommendations every day, nearly all of which are correct. How carefully would you examine the 300th recommendation? What about the 3,000th? There’s a practical productivity question here, too. If the automation is so good that almost every recommendation deserves approval, why are we requiring a person to approve every single one? Perhaps because someone decided a human checkpoint would make the process safer. But a checkpoint that rarely receives meaningful scrutiny may create an illusion of oversight without providing much protection.

The risks include several related but distinct human-factors problems.

Complacency. Repeated success can reduce the scrutiny applied to a system’s output.

Automation bias. People may place inappropriate weight on automated recommendations, overlooking contrary evidence or failing to recognize errors.

Vigilance decline. Sustained attention to repetitive monitoring tasks is difficult, particularly when meaningful events are rare.

Skill degradation. People who stop performing a task may gradually lose the proficiency needed to perform it independently.

Loss of situational awareness. Supervisors may no longer understand the developing situation well enough to detect or respond to a problem.

And there’s another organizational risk: a workplace culture that discourages disagreement with the machine. If the system is considered authoritative, rejecting its recommendation may require extra paperwork, explanation, or managerial approval. Perhaps the employee’s productivity metrics even penalize intervention. Or maybe the last person to raise their hand ended up being wrong. And they took some crap for it. Others saw that and learned a lesson.

Formal authority isn’t the same as practical agency. If employees learn that challenging automation creates trouble, delays, or embarrassment, they’ll gradually learn not to challenge it. The system may have an override mechanism, but the organization’s culture has effectively disabled it.

What Does Research Tell Us?

This concern isn’t limited to historical work on factories and aviation.

A 2024 study by Daniela Sele and Marina Chugunova, Putting a Human in the Loop: Increasing Uptake, but Decreasing Accuracy of Automated Decision-Making, examined how people interacted with automated recommendations. In their online experiment involving 292 participants, allowing human adjustment increased participants’ preference for algorithmic decision support. But the actual adjustments made by participants reduced decision accuracy on average. Another troubling finding was that participants were less likely to intervene when recommendations contained larger errors than when they contained smaller ones.

That’s not evidence that humans generally make AI systems worse. It’s one experiment involving a particular prediction task. But it challenges the convenient assumption that human involvement automatically improves decision quality.

More recently, Sue Eze’s 2026 preprint, Human-in-the-loop Isn’t a Checkbox: Designing Meaningful Intervention in Automated AI Decisions, examined the governance problem of human reviewers who formally approve decisions without the information, time, confidence, or authority to meaningfully challenge them. The paper describes this as Rubber-Stamp Risk. As a preprint, it should be distinguished from established peer-reviewed research, but the issue it identifies is highly relevant.

So it does seem there’s a recurring theme across decades of work:

Human oversight is not inherently effective. Its effectiveness depends on how the human and automated system are designed to work together.

The research doesn’t establish a universal rule that more automation necessarily means weaker human performance. System design, training, feedback, and the nature of the task matter. But it gives us plenty of reason to question simplistic claims about HITL safeguards.

What Changes With Modern AI?

The basic human-factors problems aren’t new. Industrial automation, aviation, medical equipment, and decision-support software have been dealing with them for years. Modern AI may make them harder to manage for several reasons.

1. More Abstraction Between the Human and the Work

Traditional automation often followed relatively well-defined rules or control processes. Modern AI systems can operate through multiple layers of interpretation, reasoning, retrieval, tools, and external services. Imagine a human approving an action proposed by an AI agent. That agent may have delegated research to another agent, retrieved information from several sources, called software tools, summarized intermediate results, and formulated its own recommendation. The person reviewing the final output may be several layers removed from the information and decisions that produced it.

This is why I cringe a little when I see people celebrating fleets of AI agents managed by some executive agent. Maybe these systems really are extraordinarily efficient and effective. But I wonder how many of their operators could explain what’s actually happening inside those workflows, recognize when something is going wrong, and intervene effectively. Maybe that doesn’t matter much for low-risk tasks. For higher-consequence operations, I’d want a better answer.

Meanwhile, conventional software can have complex abstraction layers too. This isn’t unique to AI. But the flexibility of language-based interfaces and agentic workflows can make those layers unusually easy to construct, extend, and hide behind a simple final answer.

Each layer may improve efficiency while increasing the distance between the reviewer and the underlying reality.

This introduces another useful distinction: a system can be technically observable without being meaningfully understandable to its human supervisor. An enormous activity log doesn’t necessarily solve the problem. The reviewer needs the relevant evidence, dependencies, uncertainties, and consequences presented in a usable form.

2. Automation Can Move From Advising to Acting

There’s a substantial difference between an AI system that drafts an email and one that sends it. Or between a system that recommends a database change and one that executes the change through a sequence of tool calls. Agentic AI can move beyond producing information into taking actions. That changes the timing and significance of human intervention.

If an agent has already updated multiple records, contacted outside systems, and triggered downstream workflows, a final human approval may be too late to serve its intended purpose. Human oversight needs to be designed around the boundaries where consequential actions occur, not merely attached to the end of a workflow.

3. The Volume of Decisions Can Grow Dramatically

AI may let a single person supervise work that previously required a large team. That’s potentially a major productivity gain. But can that same person actually exercise meaningful judgment over the resulting volume of decisions? If one manager goes from reviewing 20 consequential decisions per day to overseeing 2,000 automated decisions, something about the control model has to change. Otherwise, we may simply be scaling the appearance of oversight.

4. Mistakes Can Become Harder to Trace

An AI-generated recommendation may incorporate source errors, mistaken assumptions, outdated documents, incorrect tool results, or errors introduced by another automated component. The final answer might look entirely reasonable. A fluent explanation is not necessarily a reliable audit trail, and an AI system’s description of what it did is not a substitute for independently recorded actions and evidence. This matters when the human is expected to identify errors that aren’t apparent in the final result.

5. We May Be Accelerating Skill Decay

If an AI system takes over much of a professional’s routine work, that professional may have fewer opportunities to practice the underlying skills. Consider medical documentation. AI-powered scribes may reduce administrative burden and allow clinicians to devote more attention to patients. That’s a potentially valuable application of automation.

But there’s a separate question about how clinicians review generated notes. If a clinician verifies the record against the encounter they personally conducted, they have independent knowledge to evaluate it. If they merely accept a polished narrative, the review may be much less meaningful. Longer term, similar concerns could arise in occupations where AI takes over the routine analytical work that traditionally helps junior professionals develop expertise.

What happens when future supervisors have never spent much time doing the work they’re expected to supervise? That isn’t an argument against automation. It is an argument for thinking about how we develop and maintain expertise in increasingly automated professions. This may be particularly important in software development, where AI is increasingly capable of writing, debugging, and modifying code. This is becoming a more common concern of discussion. If junior developers spend less time doing the foundational work, how will they develop the expertise to evaluate complex systems later? Perhaps we’ll develop better training methods. But perhaps we shouldn’t assume professional judgment will emerge automatically from supervising automated output.

Human-in-the-Loop, Human-on-the-Loop, or Human Out of the Loop?

Some terminology helps here.

Human-in-the-Loop: A human participates in the decision or approval process before a specified action proceeds.
Human-on-the-Loop: A human supervises an automated system that can operate without approval at every step, with the ability to intervene when appropriate.
Human-out-of-the-Loop: The system operates without meaningful human involvement in the relevant operational decisions.

These terms aren’t always applied consistently, and real systems may combine all three approaches at different stages. More importantly, the label doesn’t tell us whether the control is effective. Someone clicking Approve 500 times a day could be less engaged than a supervisor actively monitoring a dashboard, investigating anomalies, and stopping automated processes when necessary. Likewise, a system operating without continuous human supervision may have robust engineering controls that prevent harmful actions more reliably than a fatigued reviewer could. We shouldn’t confuse the organizational label with the quality of the safeguard.

Keeping the Human Capable of Intervening

If we accept the possibility that automation can weaken the human’s ability to supervise it, then meaningful HITL needs to be an ongoing design responsibility.

Here are several principles I’d apply and I try to use as I work with these tools:

1. Give Reviewers a Real Picture

Give reviewers the information they need to understand what the automation has done, recognize exceptions, and evaluate consequences. More information isn’t necessarily better; the objective is actionable situational awareness.

2. Match Approval Points to Actual Risk

Not every action needs human authorization. A small, easily reversible, low-risk action might be safely automated. A large financial transfer, an irreversible production change, or a high-impact medical decision may justify a more substantial review. Risk-based approval thresholds, permission boundaries, and escalation rules can be better than blanket HITL requirements.

3. Preserve Independent Judgment

A reviewer who sees only the AI’s conclusion may become anchored to it. In some workflows, having the human form an initial assessment before seeing the automated recommendation could help preserve independent judgment. Other approaches might include presenting supporting evidence separately from the system’s recommendation, highlighting genuine uncertainty, or requiring explanation for especially consequential decisions. These controls should themselves be tested. Adding friction doesn’t automatically improve judgment.

4. Maintain Skills Through Practice

If humans are expected to take control during failures, they need opportunities to practice those responsibilities. Aviation has long used recurrent training, simulator scenarios, and abnormal-procedure exercises. Comparable approaches may be appropriate for other high-consequence automated workflows. This might mean periodic manual exercises, simulated failures, controlled exception handling, or regular review of decisions where automation performed poorly. Importantly, the training must reflect the kinds of problems people might actually encounter.

5. Make Intervention Practically Possible

A supervisor needs more than nominal permission to say no. Can they pause the process? Investigate the evidence? Reverse an action? Escalate a concern? Restrict further activity? And can they do so without disproportionate administrative consequences? An override button that nobody feels empowered to use isn’t a strong control.

There’s also a timing problem. Detecting that something is wrong, understanding what went wrong, and actually intervening are three different activities. A supervisor might have full authority to stop a process but not recognize the problem quickly enough. Or the system may have already executed irreversible actions before the problem becomes apparent. Meaningful oversight requires enough time for all three, not just access to a Stop button.

6. Measure Whether Oversight Works

Counting human approvals isn’t a useful measure of safeguard quality by itself.

Sometimes HITL Isn’t the Right Answer

There’s a danger of treating human oversight as the default answer to every concern about AI. But some failure modes are better addressed through engineering controls. For example, an AI agent processing routine refunds might operate within strict monetary limits, use deterministic checks for eligibility, maintain independent transaction logs, and escalate unusual cases. That might be more dependable than asking an employee to approve every refund.

Other controls could include restricted tool permissions, independent validation, transaction limits, staged execution, anomaly detection, and reversible operations. And these safeguards can complement meaningful human intervention rather than replace it. The question should always be: What specific failure are we trying to prevent, detect, or recover from? If we cannot explain how a human checkpoint addresses that failure, we should question why it’s there. HITL should be a deliberate control choice, not a generic answer to automation risk.

A Better Test for HITL

Earlier, I suggested that effective human oversight depends on awareness, capability, and control. But how do we know whether those qualities actually exist in a working system? Here’s how I’d evaluate a proposed Human-in-the-Loop process. Instead of simply asking, “Is there a human approval step?” I’d ask six questions.

Question	Dimension	What it tests
Can we recognize a problem?	Awareness	Recognition and judgment
Can we see enough to understand it?	Awareness	Situational awareness and evidence
Can we actually intervene?	Control	Authority, tools, and practical agency
Can we intervene in time?	Control	Timing and workflow design
Can we still do the job when automation fails?	Capability	Skill retention and operational readiness
Does our participation measurably improve the outcome?	All three	Actual control effectiveness

I think the fifth question is particularly important. Organizations are often reasonably good at asking whether their people have been trained. They’re less likely to ask whether the way they’ve designed the work will allow those people to remain proficient over time. A person may be qualified when an automated system is first deployed. That doesn’t establish they’ll remain equally capable after two years of mostly approving its outputs. HITL isn’t merely a feature to be implemented. It’s a capability that has to be maintained. And if the answer to the sixth question is no, that should trigger a reconsideration of the control itself.

The Real Challenge

The promise of automation has always been to reduce the amount of human effort required to accomplish something. That’s a good objective. We shouldn’t romanticize manual work, and we shouldn’t insist people perform repetitive tasks simply because doing so might keep them practiced. But there’s a fundamental tension when we simultaneously remove people from the normal operation of a system and expect them to remain expert supervisors of its most difficult exceptions.

The right response isn’t necessarily to slow everything down with more approvals. It’s to design automation, oversight, and human responsibilities together, including the means to preserve the skills and awareness needed when intervention genuinely matters.

Putting a human in the loop is easy. Keeping them capable of being there is the hard part.

And perhaps that’s the question every product manager, technology leader, and AI governance team should be asking:

If the automation fails tomorrow, will the person we’ve designated to take over actually be ready?