Supervising an AI agent after you hire it

To manage AI agents you have hired, supervise them like a new hire on a defined job: one named owner, a review level set by how much a mistake costs, clear rules for when the agent hands work back to a person, a way to stop it, and a plan for taking the job back. Supervision does not end after the trial. An agent's behavior can change when its vendor updates the model or when you change its instructions, so the checks keep running for as long as the agent does the job.

This guide is the last step of the method in how to hire an AI agent: scope, brief, trial, acceptance, supervision. It assumes you already have a brief with acceptance criteria and human checkpoints, written as in the AI agent job description guide, and trial results from an AI agent trial. It covers one agent doing one job for a small team, not a fleet of agents across a company.

The supervision plan at a glance

A supervision plan answers five questions. Write the answers into the brief's human checkpoints before the agent goes live.

Part	Question it answers	What you write down
Owner	Who is responsible for this agent's work?	One named person, with the authority to pause it
Review	What does a person check, and how much of it?	A review level for each action the agent takes
Escalation	When does the agent stop and hand work to a person?	Triggers, and the person or queue that receives the work
Change control	What happens when the agent, its instructions or its access change?	Which changes trigger a re-check against the brief
Exit	How do you take the job back?	Who stops it, what access is revoked, where the work goes

Name one owner

Every agent needs one person on your side who answers for its output. That person reads the escalations, decides when review can loosen, approves changes, and can pause the agent without asking the vendor.

Microsoft's operational standards for AI agents put this as a decision leaders have to make: "which failures count as incidents and who can stop an agent." Microsoft writes for setting standards across an organization, but the question is the same for a team of five. If nobody is named, nobody knows whose job it is to look.

The owner needs time for the job. In the first weeks of production, reviewing the agent's output is real work, and it belongs in someone's week, not on top of it.

Set review levels by risk, not by habit

Not every action needs the same review. A draft reply to a customer, a coded invoice and a payment carry different costs when they go wrong. Set the review level per action, not per agent.

OpenAI's guide to building agents suggests rating each tool an agent can use as low, medium or high risk, "based on factors like read-only vs. write access, reversibility, required account permissions, and financial impact," and using the rating to trigger checks or escalate "to a human if needed." It also lists "high-risk actions" that "should trigger human oversight until confidence in the agent's reliability grows," with examples such as "canceling user orders, authorizing large refunds, or making payments." OpenAI is a vendor; read this as its advice to builders.

The OWASP Gen AI Security Project makes the same point from the security side. Its entry on Excessive Agency (LLM06:2025) describes the risk as what "enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs" from a model, and recommends human-in-the-loop control "to require a human to approve high-impact actions before they are taken."

This guide uses three review levels. They are our recommendation, not a standard:

Level	What the person does	Use it for
Approve every output	Nothing takes effect until a person approves it	Payments, filings, deletions, messages to customers in the first weeks
Review exceptions plus a sample	The agent's flagged items are reviewed; a fixed share of the rest is checked	Work that can be corrected before it costs much, such as coding or tagging
Spot-check	A person checks a small sample on a schedule	Internal, reversible work with a stable error rate

Our recommendation: start every action one level stricter than you think it needs, and move down a level only on evidence. Pick the evidence in advance, for example "the error rate on the sample has stayed within the brief's acceptance criteria for four straight weeks." Write that rule down before production, so the decision to loosen review is not made on a good day.

Keep review where it was if you are tempted to loosen it because reviewing is tedious. The EU AI Act names this risk for the AI systems it classes as high-risk: Article 14(4)(b) asks that the people overseeing them stay aware of "the possible tendency of automatically relying or over-relying on the output," which the Act calls "automation bias." Whatever your agent's legal status, a reviewer who approves everything is not reviewing.

Limit what the agent can do in the first place

Review catches mistakes. Narrow access prevents some of them. OWASP traces Excessive Agency to three causes: "excessive functionality," "excessive permissions" and "excessive autonomy." Its mitigations include limiting the extensions an agent can call "to only the minimum necessary," limiting their permissions on other systems "to the minimum necessary," and implementing authorization "in downstream systems rather than relying on an LLM to decide if an action is allowed or not."

For supervision, this means checking access as part of the routine, not only at setup. When the owner reviews the agent each month, they confirm the agent still has only the access the brief lists. A tool that was added for a test and never removed is one of OWASP's own examples of excessive functionality.

OWASP adds two controls that limit damage without preventing it: log and monitor what the agent does "to identify where undesirable actions are taking place," and rate-limit actions so there is more time to notice through monitoring "before significant damage can occur."

Decide when the agent hands work back

Escalation is the agent giving a piece of work to a person because it should not finish it alone. OpenAI's guide describes this as letting the agent "gracefully transfer control when it can't complete a task," and names two triggers: "exceeding failure thresholds," such as when an agent "fails to understand customer intent after multiple attempts," and the high-risk actions above.

Intercom's documentation for its Fin AI Agent shows what these triggers look like in one support product. According to Intercom, Fin escalates by default when a customer "clearly asks to speak to a human," when it detects strong frustration or anger, though it may first try to resolve the issue before escalating, and when the customer "is stuck in a repetitive loop." Buyers can add escalation rules based on data, such as an order total over a set value or a "VIP_customer" attribute, and written guidance in plain language. Intercom also documents an "Ask for input" option, in beta on chat, in which Fin "holds the answer, shows a teammate Fin's draft and what Fin needs, then sends the final reply once the teammate answers."

Whatever product you use, write the escalation rules into the brief:

• Triggers. Which conditions send work to a person: low confidence, missing data, an amount over a limit, a named customer type, a request for a human.
• Receiver. The named person or queue that gets the work, with cover for when they are away.
• What travels with it. The case, what the agent already did, and why it stopped, so the person does not start from zero.
• A time limit. How long escalated work can wait before someone else picks it up.

Check that a receiver actually exists. Intercom notes that if no human routing target is configured, a mode it calls "bot only," Fin will not offer escalation over chat. An agent configured to escalate into nowhere is not escalating.

Read escalations as data. If the same kind of case keeps coming back, either the brief should take it out of scope or the agent's instructions need to cover it. If escalations stop entirely, check that the triggers still work before you take it as good news.

Watch for wrong answers, not only outages

Ordinary software monitoring tells you when a system is down. An agent can be up and still doing the job wrong. Microsoft's guidance says agent monitoring "must also detect wrong answers and unintended actions from agents that appear healthy," and that monitoring depth "should reflect the consequences of undetected failures."

For a small team, this comes down to three routine checks, scheduled by the owner:

• The sample. Check the review sample against the brief's acceptance criteria and record the error count by type.
• The escalations. Read the week's escalations: how many, which triggers, and whether any should not have been needed.
• The log. Keep a record of what the agent did, so an error found later can be traced to its cause. For high-risk AI systems under the EU AI Act, Article 26(6) requires deployers to keep the system's automatically generated logs, to the extent they control them, for "at least six months" unless other law provides otherwise. Whether your use falls under that article is a question for counsel; ask counsel which obligations apply to your role and jurisdiction.

Track cost in the same review. Microsoft suggests "cost per successful task" as a measure "when outcomes are comparable." Compare it in the vendor's pricing unit; how AI agents are priced explains the units.

Re-check after every change that matters

An agent that passed its trial can behave differently later. Microsoft's guidance notes that "model replacements and instruction updates can change agent outputs" and that "expanded permissions can introduce additional risks," and recommends that changes which can "materially affect agent behavior should trigger re-evaluation" before they reach production.

In the brief, list the changes that send the agent back to a short re-check against its acceptance criteria:

• the vendor changes the underlying model or releases a new agent version
• you change the agent's instructions, knowledge sources or escalation rules
• the agent gets new access or a new tool
• the job itself changes: a new product line, a new supplier, a new policy

Ask the vendor in writing how they notify you of model and version changes. If they do not, schedule a re-check on a fixed date instead.

Plan the handover back before you need it

Every supervision plan needs an exit. Someone on your side must be able to stop the agent and take the job back without the vendor's help. For high-risk AI systems, Article 14(4) of the EU AI Act requires that the people overseeing them can "disregard, override or reverse" the output and can interrupt the system "through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state." Our recommendation for any agent: ask for the same ability before you sign.

Write the handover steps down:

1. Stop. Who pauses the agent, and where in the product they do it.
2. Contain. Which in-flight work is held, and who checks what the agent did since the last good review.
3. Reassign. Which person or team takes the work, and what they need: the queue, the log, the open escalations.
4. Revoke. Remove the agent's access. Microsoft's guidance on retiring an agent says retirement "should disable agent execution and revoke associated permissions."
5. Restart, or not. Microsoft recommends that restoring an agent after an incident depend "on evidence that the underlying issue is resolved." Our recommendation: rerun the relevant part of the trial before the agent resumes.

1. 1

   Stop

   Who pauses the agent, and where

2. 2

   Contain

   Hold in-flight work and check what the agent did

3. 3

   Reassign

   Who takes the work, with the queue, log and open escalations

4. 4

   Revoke

   Remove the agent's access

5. 5

   Restart, or not

   Rerun the relevant part of the trial first

Practice the handover once during the first month. A pause button nobody has pressed is an assumption, not a control.

A worked example

Example only; not a real buyer. Assume, for this example, the numbers below. A support manager at a 40-person software company runs an agent on tier-1 support tickets. After a trial on 200 past tickets, the agent drafts replies to password, billing-date and plan-limit questions.

• Owner. The support manager, who can pause the agent from the help desk.
• Review. Weeks 1–4: a person approves every reply before it is sent. From week 5, if errors on the reviewed replies stayed within the brief's limit of at most 2 wrong answers per 100, the agent sends replies on its own and a person checks 20 sent replies a day. Refunds stay at "approve every output."
• Escalation. The agent hands over when the customer asks for a person, mentions a refund, or repeats the same question. Handovers go to the tier-2 queue with the agent's summary attached. Escalated tickets older than 4 business hours go to the manager.
• Change control. When the vendor announces a model update, the manager reruns 50 of the trial tickets before switching it on.
• Exit. In week 2 the manager pauses the agent for one hour to test the handover. Tickets route to the team's normal queue, and the agent's help-desk role is confirmed as the only access to revoke.

The same plan, with different review levels, fits ticket triage, accounts payable and document processing.

Write the plan into the brief

The supervision plan lives in the brief's human checkpoints and constraints. Copy the template and fill in those two fields with your owner, review levels, escalation triggers and exit steps.

Questions buyers ask

How long do I have to review every output? No standard source sets a period. Decide the evidence you need in advance, such as a stable error rate on your sample for a set number of weeks, and loosen review only when it is met.

Can the vendor supervise the agent for me? A vendor or operator can do the day-to-day checks if the brief says so. The owner on your side still decides what counts as acceptable work and keeps the ability to stop the agent.

Who is responsible when the agent gets something wrong? Ask counsel which obligations apply to your role and jurisdiction. Put the provider's obligations, such as fixing errors, notifying you of model changes and handling your data, in writing before production.

What if the agent escalates too much? Look at which triggers fire. Frequent escalations on one case type suggest narrowing the scope or improving the instructions. Do not loosen a trigger only to bring the count down.

Sources

1. 1
   A practical guide to building agents

   OpenAIVendorAccessed Nov 14, 2026

2. 2
   Operational standards for AI agents (Cloud Adoption Framework)

   MicrosoftVendorAccessed Nov 14, 2026

3. 3
   Manage Fin AI Agent's escalation guidance and rules

   IntercomPrimaryAccessed Nov 14, 2026

4. 4
   LLM06:2025 Excessive Agency

   OWASP Gen AI Security ProjectPrimaryAccessed Nov 14, 2026

5. 5
   Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal L, 12.7.2024

   Publications Office of the European UnionPrimaryAccessed Nov 14, 2026

6. 6
   AI Act Service Desk: Article 14: Human oversight

   European CommissionPrimaryAccessed Nov 14, 2026

7. 7
   AI Act Service Desk: Article 26: Obligations of deployers of high-risk AI systems

   European CommissionPrimaryAccessed Nov 14, 2026