Running a trial with an AI agent on your own work before you hire it

To test an AI agent before buying, run a short trial on a sample of your own past work that the vendor has not seen, review every output before it takes effect, and score the results against the acceptance criteria in your brief. Agree the terms in writing first: what the agent can access, what happens to your data, who reviews, how long it runs and what result counts as a pass. We recommend paying for the trial, so that both sides treat it as real work. A demo shows an agent on the vendor's chosen cases. A trial shows what it does on yours, and it ends in a number you can accept or reject.

This guide is step 3 of the method in how to hire an AI agent. It assumes you have already scoped one job and written a brief with acceptance criteria; if not, start with the AI agent job description guide. Nothing here depends on a particular product.

What a trial is for

A trial answers one question: does this agent meet your brief on your work, at your volume, under the review you will actually run? It is not a feature tour, and it is not a pilot that drifts into production without a decision.

The method below borrows from how agent builders test their own systems. Anthropic, which builds models used in agents, describes an evaluation as giving an AI "an input" and applying "grading logic to its output to measure success." It distinguishes the transcript (what the agent said and did) from the outcome, "the final state in the environment at the end of the trial." Its example: a flight-booking agent might say the flight is booked, "but the outcome is whether a reservation exists." For a buyer, that distinction is the core of the trial. Score what changed in your systems, not what the agent reports.

Public-sector buyers are told to test the same way. In its April 2026 report on federal AI acquisitions, GAO summarizes OMB Memorandum M-25-22 as stating that agencies "should test proposed AI solutions to understand their capabilities and limitations before awarding contracts" and should be able to "regularly monitor and evaluate an AI system" afterward. That guidance is for U.S. federal agencies, not private buyers, but its testing advice may be useful to private buyers too.

Before the trial: agree the terms

Write the terms down before any data moves. A one-page note is enough. It covers:

Term	What to settle
Scope	The one job from your brief, and what is out of scope
Sample	How many units of work, which period, and who picks them
Access	The exact accounts and permissions the agent gets, and when they are revoked
Data	Where your data is processed and stored, whether it is used for anything beyond the trial, and how it is deleted afterward
Review	Who on your side checks each output, and who on the vendor's side fixes problems
Length	Start and end dates
Price	What the trial costs, in the vendor's pricing unit
Pass bar	The acceptance criteria from your brief, unchanged, and what happens on a pass or a fail

Two of these deserve care.

Access. Give the agent only what the job needs for the trial. The OWASP Gen AI Security Project lists Excessive Agency as LLM06:2025 and traces it to three causes: excessive functionality, excessive permissions and excessive autonomy. Its mitigations include limiting the permissions an agent is granted "to the minimum necessary" and using "human-in-the-loop control to require a human to approve high-impact actions before they are taken." Its example of excessive permissions is an extension meant to read data that connects with an identity that can also update, insert and delete records. In a trial, read access and draft rights are usually enough. Revoke them on the end date.

Data. GAO's summary of M-25-22 lists contract terms on "use of government data, intellectual property rights, privacy, and vendor lock-in" for agencies to address at award. Swap "government data" for your own and the list fits a private trial. Ask counsel which data protection obligations apply to the personal or customer data in your sample.

Why pay. Our recommendation, not a sourced rule: offer to pay for the trial at the vendor's normal unit price, capped at the sample size. Paying does not by itself secure the vendor's attention, a named fixer or written terms; negotiate those whether or not the trial is paid. It also gives you a real cost-per-unit figure for the comparison in how AI agents are priced. We found no source on how common paid trials are, so we make no claim about what vendors usually offer.

Build the sample

The sample is the most important decision in the trial. Build it from past work you already know the right answer to.

Hold it back. According to GAO's summary, M-25-22 states that agencies should use their own validation and testing data for independent evaluations, and that this data "should not be accessible to the vendor" and should be "as similar as possible" to the data the system will see when deployed. Apply both rules. Do not send the vendor the sample in advance, and draw it from recent, ordinary work rather than cleaned-up examples. If the vendor tunes the agent during the trial, tune on different items from the ones you score.

Size it to your volume. No primary source gives buyers a required sample size. Anthropic's own advice to teams building evaluations is that "20-50 simple tasks drawn from real failures is a great start" in early development, when changes have large effects; it says more mature agents "may need larger, more difficult evals." Read that as a vendor's rule of thumb for builders, not a standard for buyers. For a first trial, a sample in the tens, drawn from about one month of work, is a practical starting point. Example: 50 invoices from last month.

Include the hard cases on purpose. Put in the units your team finds awkward: the scanned invoice, the angry customer, the lead with no website, the ticket that belongs to no queue. Anthropic recommends testing "both the cases where a behavior should occur and where it shouldn't." For a buyer, that means including items where the right action is to escalate, decline or ask, not only items where the right action is to act.

Write the answer key. For each item, record the correct outcome before the trial starts. Anthropic's test of a good task is one where "two domain experts would independently reach the same pass/fail verdict." If two people on your team disagree about the right answer for an item, fix the answer key or drop the item; otherwise the disagreement shows up as noise in the score.

Run it

Run the agent in the setup you would use in production: the same systems, the same access, the same review. Anthropic notes that it is "essential that the agent in the eval functions roughly the same as the agent used in production." A trial in the vendor's demo environment tests the demo.

During the run:

1. Review every output before it takes effect. Nothing the agent produces reaches a customer, a ledger or a supplier without a person approving it. This is the trial's human checkpoint, and it is also what keeps a bad trial cheap.
2. Score outcomes, not reports. Check the record in your system, the queue the ticket landed in, the field that was updated.
3. Log every intervention. Each time a person had to correct, finish or redo a unit, record which item, what was wrong and how long the fix took.
4. Record time and cost per unit, in the vendor's pricing unit (per seat, per task, per resolution, per minute).
5. Read the transcripts of failures. Anthropic writes that "when a task fails, the transcript tells you whether the agent made a genuine mistake" or whether the grading was wrong. A failure caused by your unclear brief is a brief problem; a failure on a clear item is an agent problem.

1. 1

   Review every output

   Before it takes effect

2. 2

   Score outcomes, not reports

   Check the record in your system

3. 3

   Log every intervention

   Which item, what was wrong, how long the fix took

4. 4

   Record time and cost per unit

   In the vendor's pricing unit

5. 5

   Read the transcripts of failures

   Brief problem or agent problem

Run some items more than once. Agent output can vary between runs. Anthropic runs "multiple trials" per task for that reason and describes a measure, pass^k, of the probability that all k runs succeed. Its arithmetic: an agent with a 75% success rate per run passes 3 runs in a row with probability (0.75)³ ≈ 42%. Anthropic says this "especially matters for customer-facing agents where users expect reliable behavior every time." For a buyer, rerunning a handful of items two or three times shows whether a pass was luck.

Score it and decide

Score each acceptance criterion from your brief as pass or fail on the trial data. Then add up:

• Error rate by type, against the answer key.
• Escalation behavior. Did the agent hand over the items it should have, and only those?
• Interventions per 100 units, and the time they took.
• Cost per unit at the trial's measured volume.
• Consistency on the rerun items.

The decision has three honest outcomes, the same as in the hiring guide: accept, narrow the scope, or reject. Narrow the scope when failures cluster in a recognizable subset (foreign-currency invoices, refund requests), take that subset out of the job, and rerun the trial on the narrower scope with a fresh sample.

Do not move the bar after seeing the results. If you change a criterion, write down why. "The agent could not meet it" is not a reason to change the criterion.

A worked example

Example only; not a real buyer or product. Assume, for this example, that a support manager at a software company wants an agent for support ticket triage: tag each new ticket with a category and priority and route it to one of five queues. Last month the team handled about 3,000 tickets.

• Terms. Two weeks. Read access to the help desk and permission to suggest tags and queues; no permission to reply to customers or close tickets. The vendor confirms in writing where ticket data is processed and that it is deleted after the trial. The trial is paid at the vendor's per-ticket price for the sample only.
• Sample. 60 tickets from last month, held back from the vendor: 45 ordinary tickets, 10 the team found hard to route, and 5 that should be escalated to a person (a legal threat, a security report, an angry enterprise customer). The answer key is the queue and priority two team leads agreed on.
• Brief's pass bar. At least 55 of 60 routed to the right queue; all 5 escalation tickets flagged; no ticket marked low priority that the answer key marks urgent.
• Run. A team lead approves each suggestion before it applies. Ten tickets are rerun three times each.
• Result. 54 of 60 routed correctly; 4 of 5 escalations flagged; one urgent ticket marked low. Five of the six misroutes involve billing tickets that mention a product feature. Two of the ten rerun tickets got different queues on different runs.
• Decision. Fail as scoped. The missed escalation and the urgent-marked-low ticket are disqualifying on their own. The manager narrows the scope to non-billing tickets, keeps escalation detection under human review, and reruns with a fresh 60-ticket sample.

The same steps apply to accounts payable, document processing, tier-1 support and lead research and enrichment: hold back real work, review every output, score against the brief.

After the trial

A pass is permission to start supervised production at the trial's volume, not at full volume. Keep the trial sample: rerun it whenever the vendor changes the model or the agent's setup, and compare the score. Anthropic describes keeping tasks that an agent passes reliably as a regression suite that "is run continuously to catch any drift." GAO's report notes concerns, raised by members of Congress and others, that "model performance may degrade over time." Revoke the trial access on the end date, then grant production access separately, as narrowly as the brief allows. Review cadence, escalation and handover are covered in how to supervise an AI agent.

Questions buyers ask

How long should a trial run? Long enough to cover a representative slice of your work, including the awkward cases. No primary source sets a length; the two weeks in the example is an example.

Can I use a free trial instead? You can, but check what it runs on. A free tier that runs on sample data or a sandbox does not test your work. Whatever the price, the sample, the answer key and the pass bar stay the same.

What if the vendor wants to see the sample first? Share the brief and a few examples of the work, not the scored items. GAO's summary of M-25-22 is explicit that validation data "should not be accessible to the vendor."

What if I can't build an answer key? Then the output is hard to check, and the job may not suit an agent yet. Have two people grade a small set independently; if they disagree often, tighten the brief before you trial anything.

Sources

1. 1
   Demystifying evals for AI agents

   AnthropicVendorAccessed Nov 14, 2026

2. 2
   GAO-26-107859, Artificial Intelligence Acquisitions: Agencies Should Collect and Apply Lessons Learned to Improve Future Procurements (April 2026)

   U.S. Government Accountability OfficePrimaryAccessed Nov 14, 2026

3. 3
   LLM06:2025 Excessive Agency

   OWASP Gen AI Security ProjectPrimaryAccessed Nov 14, 2026