AI-first company

Governance and risk · Adopt

Set the limits before the tool acts

Before an AI tool sets a price, sends a message or takes a step for you, write down what it must never do, how far it may go, and what needs a person. Build those limits into the tool, then test inside them.

By
Last reviewed
Corrections
Report an error
When Do next. Once your first uses of AI are running.
Effort Low (days, one person)
Evidence Common practice

One measured case, an online retailer, fixed its pricing tool’s limits before any test. Two more show the split between tool and person decided in advance, and a regulator set the limits for an AI-run law firm. No case or study compares a tool run with limits against one run without.

Fits
Industries All industries
In the guide What fails

What it is

Before you switch on an AI tool that acts, write three short lists:

  • Never: what the tool must not do in any case.
  • How far: the range the tool may work in, with a number.
  • Needs a person: the actions that wait for someone’s approval.

Then build the lists into the tool, so that the tool cannot go outside them.

Some AI tools only draft. Others act: they recommend a price that goes on the site, send a reminder to a client, answer a customer, decide who gets through to the next step.

What the evidence shows

One case is the main basis for this advice. Two others show a similar split, and a regulator has set limits of the same kind. No study has tested it.

  • The limits came before the test. Rue La La, an online fashion retailer, had a tool built that recommends prices for products the retailer has never sold. The retailer set the limits first. The old price was the floor, so the tool could recommend a rise or no change. A rise could be no more than $15 or 15%, whichever was larger, and never above a set discount from the list price. The retailer accepts a rise only if the price is still lower than on other websites.

  • Inside those limits the tool still made money. In a five-month test, researchers estimated that revenue on the tested products rose by about 9.7%.

  • The limits answered the managers’ worry. The managers feared that higher prices would cut sales and leave unsold stock. The test was run inside the limits, and its result was what won their approval.

  • Limits kept the tool close to what it knew. The tool predicts from past sales. The researchers write that restricting the prices the tool could choose helped avoid predictions for situations the tool had no data on.

  • A split decided in advance. A two-person advisory firm lets AI send document reminders to clients unaided. About 70% of reminders go out with nobody stepping in. The adviser approves important letters, and the system raises an alert when a person is needed. A recruitment firm let an AI voice agent run job interviews. Every hiring decision stayed with a human recruiter.

  • A regulator wrote the limits into an authorization. England’s Solicitors Regulation Authority authorized Garfield.Law, a law firm whose services for small debt claims are delivered by AI, with conditions. The system may not propose case law. It acts only when the client approves each step. Named solicitors stay accountable for everything it produces.

  • Builders keep the steps short. In a study of teams that build AI agents, 68% of agents in production ran at most 10 steps before a person stepped in.

A company answers for what its tool does. A tribunal ordered Air Canada to pay a customer who relied on a wrong answer from its website chatbot.

How to do it

  1. Write down what the tool does. One line. Say whether it recommends or acts. Rue La La’s tool recommends prices to its merchants.
  2. List what it must never do. Ask what would lose you money, customers or trust. Rue La La’s answers: never price below the old price, never above the set discount from the list price, never above other websites.
  3. Put a number on how far it may go. Rue La La capped a rise at $15 or 15%. For a tool that sends messages, the number might be how many reminders a client gets. For a tool that spends money, the number might be the amount.
  4. Sort every action into three groups. The tool does it unaided, a person approves it first, or it is never done. The advisory firm put reminders in the first group and important letters in the second.
  5. Build the limits into the tool. Rue La La’s tool could only choose from a set of allowed prices. A limit that lives in a document depends on someone remembering it. See what happened to a written rule at a law firm.
  6. Test inside the limits, and read the result by group. Rue La La’s test showed a gain overall and a loss on its cheapest products.
  7. Change the limits on what the test shows. After the test, Rue La La cut the largest allowed rise to $5 for products priced under $50.
  8. Watch closely before it runs unattended. Rue La La’s tool went fully automatic only after a period of close monitoring of its output.
  9. Review the limits when prices, policies or products change. Name the person who does it. See name one accountable AI owner.

What to measure

  • How often the tool reaches a limit. If the tool reaches a limit most of the time, the limit is making the decision.
  • Results by group of products or customers, as well as the average.
  • The share of actions the tool takes with nobody stepping in, and what people changed in the rest.

Where it fails

  • One limit for everything. Rue La La’s cap worked for mid-priced and expensive products. On the cheapest ones, higher prices cut sales enough that revenue probably fell. Set limits by group where the groups differ.
  • Limits written down and not built in. People forget rules, and a tool does not read them.
  • A second AI as the supervisor. In Anthropic’s own experiment, an AI ran a small office shop and a second AI acted as its manager. The manager approved lenient customer requests about eight times as often as it refused them. Requiring the shop AI to check prices and delivery times helped more than the extra layer did. Written procedures are limits. Another AI on top is not.
  • Limits set once. A limit that fitted last year’s prices or policy may be wrong this year.
  • Setting limits after a failure. By then a customer has the wrong price or the wrong answer, and you answer for it.

By company size

  • Micro and small: three short lists on one page are enough. The advisory firm above has two people.
  • Medium: the person who owns the process writes the lists, and whoever sets up the tool builds them in. Test for a set period before the tool runs unattended.
  • Mid-market and enterprise: limits are part of the specification for any tool that acts, with a log of each time a limit is reached and a named owner for each tool.

Company cases

Companies that did this

Sources

No published study tests this practice directly, so it is rated Common practice. The rating rests on the company cases above and on the sources below.

  1. Solicitors Regulation Authority (2025). Garfield.Law authorization. sra.org.uk
  2. Pan, Arabzadeh, Cogo and others (2026). Measuring agents in production. ICML 2026. arXiv 2512.04123. arxiv.org
  3. Anthropic with Andon Labs (18 December 2025). Project Vend: Phase Two. A vendor's own experiment. anthropic.com