At a glance
- Source
- Vaccaro, Almaatouq and Malone (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293–2303. doi:10.1038/s41562-024-02024-1
- How it was done
- The authors searched three research databases for experiments published from January 2020 to June 2023. Each experiment had to measure humans alone, AI alone, and the two together. Meta-analysis. It pools the results of many earlier studies into one estimate.
- Who took part
- 106 experiments, giving 370 separate comparisons.
- Where
- Experiments from many countries and many kinds of task.
“We found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone.”
What it found
- On average, the combination lost. Human-plus-AI teams performed worse than the better of the human alone or the AI alone. The gap was small (a standardized difference of −0.23, a measure that puts different tests on one scale, with a 95% range from −0.39 to −0.07).
- Decision tasks lost. Where the job was to make a decision, such as judging or classifying a case, adding a human to the AI or the AI to a human made results worse.
- Creation tasks gained. Where the job was to create content, the combination did better, by more than chance.
- Explanations made no difference on average. Showing people an explanation of the AI’s output, or how confident the AI was, did not change, beyond chance, how the human-plus-AI team performed.
- It matters who is better alone. When the human alone beat the AI alone, the combination beat both. When the AI alone beat the human, the combination did worse than the AI.
When the result holds
The comparison is strict. It asks whether the team beats the better of the two, not whether AI helps a person. A person with AI often does better than the same person without it. That is a different question.
Limits
- The authors note possible publication bias: experiments with striking results are more likely to be published.
- The experiments were published between 2020 and mid-2023, so many of them predate current AI chat tools.
Practices that use this study
This study supports:
This study limits or contradicts:
- Keep a person checking anything that goes to a customer or carries legal weight Before AI output leaves the company or is relied on in a decision, a named person checks it against real sources and signs it off. Make the check a step in the process, with time allowed for it. Do first · Effort: medium (weeks, a few people)
- Make the AI show what its answer is based on A score or an estimate with no reason attached is hard to judge. Choose and set up AI tools so that each answer comes with what it rests on: the marked part of the image, the similar past cases, the passage in the document. Do next · Effort: medium (weeks, a few people)
Assessments linked to this study
- How far do staff trust one named AI system? Trust in Automation Scale, short form (S-TIAS) The people who use that system · 3 questions, or 12 in the long form Free to use and adapt
- In which roles could generative AI take over tasks, and in which could it assist? ILO index of occupational exposure to generative AI Nobody is surveyed · You match your job titles to a standard list of occupations Free to use and adapt
- Which of our own tasks could AI do or help with? TPAH task analysis Nobody is surveyed. Process owners check the results · An analysis from a description of the company and its roles Paid service