AI-first company

Process redesign · Redesign

For each kind of task, test whether a person with AI beats either one alone

Do not assume that adding a person to AI, or AI to a person, gives the best result. Test it for each type of task before you settle the process.

By
Last reviewed
Corrections
Report an error
When Do next. Once your first uses of AI are running.
Effort Medium (weeks, a few people)
Evidence Strong Context-dependent

A study that combined 106 experiments, and a separate experiment with 758 consultants, reach the same result: it depends on the task. Many of the 106 were tasks set in a laboratory, not day-to-day business.

Fits
Industries All industries
In the guide EnterprisesWhat fails

What it is

For each type of task, compare three options: a person alone, AI alone, and a person working with AI. Then choose. Many companies treat “AI with a person checking” as the safe default. The evidence says it is not always the best set-up.

What the evidence shows

A study that combined the results of 106 experiments compared all three options. On average, a person working with AI did worse than the better of the two working alone.

The average hides a pattern:

  • Decisions. On tasks such as judging or classifying a case, a person working with AI did worse.
  • Creating content. On tasks such as drafting, a person working with AI did better.
  • Who is better alone. When people alone beat the AI, adding AI helped. When the AI alone beat people, adding a person made results worse.

An experiment with 758 consultants shows the same split inside one job. On 18 tasks the AI handled well, consultants using AI completed 12.2% more tasks and worked 25.1% faster. On one task chosen because the AI did it badly, they were 19 percentage points less likely to reach the correct answer.

Newer studies add four points:

  • Open-ended advice with no check can hurt. In a randomised trial with 640 small-business owners in Kenya, a GPT-4 business mentor made no difference on average. Owners who were doing poorly at the start did nearly 10% worse with it. Nobody checked the advice against a standard before they acted on it.
  • AI with a route to a person can beat people alone. In randomised experiments on a large online retail platform, an AI service that passed harder questions to a person sold 11.5% more than human agents alone.
  • People sort their own tasks. In two randomised trials of AI scribes, doctors given the tools used them in about a third of their work: in 33.5% and 29.5% of visits for the two tools in one trial, and for 38% of notes in the other. The set-up you choose is not always the one people use.
  • Success falls as tasks get longer. In usage data from its own business customers, Anthropic, which sells AI models, estimates that success fell from about 60% for tasks a person would do in under an hour to about 45% for tasks of five hours or more. A model judged the success.

People can learn to divide the work better. In an experiment, people who were taught how AI works handed over the right tasks more often, and their results improved.

How to do it

  1. List the types of task in the process, not the job titles.
  2. For each type, take a sample of real past cases where you know the right answer or can judge quality.
  3. Run the sample three ways: person alone, AI alone, person with AI.
  4. Compare quality first, then time.
  5. Where AI alone is clearly better, do not add a routine human edit. Use spot checks instead.
  6. Where people alone are better, let them use AI as a helper.
  7. Repeat the test when the AI model or the task changes.

What to measure

  • Quality or accuracy for each of the three set-ups, on the same cases.
  • Time per case.
  • How often the human reviewer changes a correct AI answer into a wrong one, and the reverse.

Where it fails

  • You cannot test quality. If nobody can say what a good answer is, you cannot run the comparison. Fix that first.
  • Law or duty requires a person. Some decisions must have a human who answers for them, whatever the test shows. The test then tells you how much weight to give the AI’s suggestion.
  • What AI does well keeps changing. A result holds for one AI model at one time. In the consultants’ study, tasks that looked equally hard fell on different sides of the line between what the AI did well and what it did badly.

By company size

  • Micro and small: you will not run a formal test. Take 20 past cases and compare by hand. That is enough to catch the worst mismatches.
  • Mid-market and enterprise: make the three-way test a standard step before any AI use is rolled out more widely.

Company cases

Companies that did this

Companies that came round to it

Sources

  1. Vaccaro, Almaatouq and Malone (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293–2303. What this study did and found
  2. Dell’Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and Lakhani (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science, 37(2), 403–423. What this study did and found
  3. Pinski, Adam and Benlian (2023). AI Knowledge: Improving AI Delegation through Human Enablement. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. What this study did and found
  4. Otis, Clarke, Delecourt, Holtz and Koning (2026). The Uneven Impact of Generative AI on Entrepreneurial Performance: Evidence from a Field Experiment in Kenya. Management Science (published 10 July 2026). What this study did and found
  5. Fang, Yuan, Zhang, Donati and Sarvary (2025). Generative AI and Sales Productivity: Field Experiments in Online Retail. arXiv 2510.12049, version 6 (29 June 2026). What this study did and found
  6. Lukac, Turner, Vangala, Chin, Khalili, Shih, Sarkisian, Cheng and Mafi (2025). Ambient AI Scribes in Clinical Practice: A Randomized Trial. NEJM AI, 2(12). What this study did and found
  7. Anthropic (January 2026). Anthropic Economic Index report: Economic primitives. Anthropic sells AI models; the figures come from its own customers' usage, classified by its own model. anthropic.com