AI-first company

Governance and risk · Adopt

Keep a person checking anything that goes to a customer or carries legal weight

Before AI output leaves the company or is relied on in a decision, a named person checks it against real sources and signs it off. Make the check a step in the process, with time allowed for it.

By
Last reviewed
Corrections
Report an error
When Do first. Before you start with AI, or as you start.
Effort Medium (weeks, a few people)
Evidence Emerging Context-dependent

In nine documented cases, most from courts and regulators, unchecked AI output went out and the company paid for it. Courts have dealt with more than 2,000 such decisions. None compares companies with and without a review step, and studies show that a human check does not help on every kind of task.

Fits
Industries All industries
In the guide Micro and small businessesWhat fails

What it is

Before AI output is used, have a named person check it and take responsibility for it. Make this a fixed step for two kinds of output.

  • Anything that goes to a customer or outside the company: emails, quotes, reports, advice, chatbot answers, published text.
  • Anything with legal or financial weight: contracts, court and regulatory filings, hiring and firing decisions, credit decisions, accounts.

Checking means comparing the output with something real: the source document, the database, the policy, the facts of the case. Reading it for tone is not a check.

A check may also become a condition of your insurance. In the United States, trade press reports that standard liability policy wording can now exclude losses that come from generative AI. See know the rules that apply.

What the evidence shows

The documented cases listed under “Company cases” below are the basis for this advice. In each one, AI output went out with nobody checking it against a source.

The cases are not oddities. As of 16 October 2026, one researcher’s database lists 2,199 court decisions involving AI-invented material: 1,266 involving people without a lawyer, 871 involving lawyers and 32 involving judges.

  • Made-up sources reached courts. A New York firm filed six court decisions that did not exist. A ten-person English firm filed 18. A large US firm filed eight, from its own in-house AI tool. Another filed five. Each time, looking the sources up would have caught the error.
  • A second signature did not help. In the large US firm’s case, two more lawyers signed the document without reading it. The court fined them too.
  • A wrong report reached a court and a client. A child protection report drafted with ChatGPT described a serious risk to a child as a strength. A consulting report had to be reissued twice.
  • A chatbot’s answer bound the company. A tribunal ordered Air Canada to pay a customer who relied on its chatbot’s wrong answer.
  • Staff treated an alert as a fact. At Rite Aid, employees accused shoppers because a facial recognition system flagged them. The system was often wrong.
  • Where the split was designed, it worked. A recruitment firm let AI run interviews and kept every hiring decision with a human recruiter. Job offers and staff retention rose in a study where applicants were assigned at random to an AI or a human interviewer.

Studies explain the limits:

  • AI is confidently wrong on some tasks. In an experiment with 758 consultants, people using AI were 19 percentage points less likely to reach the right answer on a task the AI did badly.
  • Having a person involved is no guarantee. Across 106 experiments, a human working with AI did worse on average than the better of the two alone. A reviewer who only skims adds little. The check has to be a real comparison with a source, done by someone who knows the subject.
  • The person checking may be losing the skill the check needs. In a survey of 319 knowledge workers, those with more confidence in AI reported less critical thinking about its output. The answers are self-reported, and the authors work for Microsoft, which sells AI tools. In a study at four endoscopy centres in Poland, doctors found fewer growths without the AI three months after it was brought in.
  • A check can be sent to where the risk is. In a Swedish breast-screening trial with 105,934 women, AI decided which scans got one reader and which got two. More cancers were found, 6.4 against 5.0 per 1,000 women, and the radiologists’ reading work fell by 44.2%. A person did not check every scan the same way.

How to do it

  1. List what goes out. Write down every kind of output where AI is used and which of them reach a customer or carry legal weight.
  2. Name the checker for each. One person, by name or role. The checker must know the subject well enough to spot an error.
  3. Say what to check against. For facts and figures: the source document. For citations and references: look each one up in a real database. For chatbot answers: the current policy or price list.
  4. Put the check in the process. Add a line to the sign-off form, the filing checklist or the publishing workflow. One of the law firms above adopted a check before every court filing: confirm that each citation exists, is accurate and is relevant.
  5. Make the signature mean something. Whoever signs has read the document. A court called this duty one that cannot be handed to someone else.
  6. Check incoming material too. In the English case the AI research came from the client. Apply the same check to drafts from clients, suppliers and freelancers, and to AI advice you act on yourself. In the Kenya trial, owners acted alone on an AI adviser’s suggestions, and the weakest did worse. See check AI advice before you act on it.
  7. For chatbots, sample. This includes the AI inside your ad platform. The UK advertising regulator holds the advertiser responsible for ads put together by automated tools. Nobody can read every answer before it is sent. Read a sample each week, test the bot after every policy or price change, and give customers a way to reach a person.
  8. Send the checking effort to where the risk is. Check the high-risk outputs most closely, as the screening trial did. Do not spread the same light check over everything.
  9. Allow the time. See budget for the work AI creates.

What to measure

  • The share of outgoing AI-assisted work with a recorded check.
  • Errors caught by the check, per month. Keep examples for training.
  • Errors found after the work went out, and who found them.
  • For chatbots: the share of sampled answers that match current policy.

Where it fails

  • The check becomes a rubber stamp. People approve without reading. This is an effect of the tool as well as of discipline: checking weakens as trust in the tool grows. Sample approvals, and test checkers on cases with planted errors.
  • The checker cannot tell. A junior person reviewing work outside their knowledge will pass a fluent error. In the English judgment, a pupil barrister said none of her written work was checked, and the law centre that instructed her said it had never checked counsel’s citations. Each side relied on the other.
  • An AI checking an AI. In Anthropic’s own experiment, an AI ran a small office shop and a second AI acted as its manager. The manager shared the shop AI’s blind spots and approved lenient customer requests about eight times as often as it refused them. Requiring the shop AI to check prices and delivery times helped. A check works only if it fails in different ways from what it checks.
  • Checking everything. If every internal note needs sign-off, the gain from AI disappears and people route around the rule. Keep the step for what goes out and what carries weight.
  • Trusting your own tool more. The eight made-up cases above came from a system the firm had built itself.
  • Guidance without a step. The English court said that issuing guidance “on its own is insufficient”.

By company size

  • Micro and small: the owner or a senior colleague reads everything that goes to a customer. A court applied the same duty to a firm of ten people.
  • Medium: a named checker for each kind of output, and a line on the sign-off checklist.
  • Mid-market and enterprise: the check is built into document and publishing systems, with a sample of approvals checked. For chatbots and automated decisions, a team owns testing and monitoring.

Company cases

Companies that did this

What happened without it

Companies that came round to it

Sources

  1. Dell’Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and Lakhani (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science, 37(2), 403–423. What this study did and found
  2. Hernström, Josefsson, Sartor, Schmidt, Larsson, Hofvind, Andersson, Rosso, Hagberg and Lång (2025). Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI): a randomised, controlled, parallel-group, non-inferiority, single-blinded, screening accuracy study. The Lancet Digital Health, 7(3), e175–e183. What this study did and found
  3. Vaccaro, Almaatouq and Malone (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293–2303. Limits or contradicts this practice. What this study did and found
  4. Lee, Sarkar, Tankelevitch, Drosos, Rintel, Banks and Wilson (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Article 1121. Limits or contradicts this practice. What this study did and found
  5. Budzyń, Romańczyk, Kitala and colleagues (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology & Hepatology, 10(10), 896–903. Limits or contradicts this practice. What this study did and found
  6. R (Ayinde) v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin). High Court of England and Wales, 6 June 2025. judiciary.uk
  7. European Union. AI Act, Regulation (EU) 2024/1689, Article 26 (obligations of deployers of high-risk AI systems). Text on the European Commission’s AI Act Service Desk. ai-act-service-desk.ec.europa.eu
  8. Charlotin, D. AI Hallucination Cases database (counts as of 16 October 2026). damiencharlotin.com
  9. Anthropic with Andon Labs (18 December 2025). Project Vend: Phase Two. A vendor's own experiment. anthropic.com
  10. Advertising Standards Authority (11 June 2026). AI and Deepfakes: Four Things Advertisers Need to Know Before They Hit 'Run'. asa.org.uk
  11. Insurance Business (16 September 2026). ISO's generative AI exclusion is already on thousands of CGL policies. Trade press. insurancebusinessmag.com