AI-first company

Data and tools · Adopt

Make the AI show what its answer is based on

A score or an estimate with no reason attached is hard to judge. Choose and set up AI tools so that each answer comes with what it rests on: the marked part of the image, the similar past cases, the passage in the document.

By
Last reviewed
Corrections
Report an error
When Do next. Once your first uses of AI are running.
Effort Medium (weeks, a few people)
Evidence Moderate Contested

Two experiments, in one paper checked by other researchers before publication, found that experts who saw what the AI had reacted to decided better than experts who saw only its score. A study that combined 106 earlier experiments found that explanations made no difference on average.

Fits
Industries All industries
In the guide Tools and systems by size

What it is

Change what the person sees when an AI tool gives an answer. Beside each answer, have the tool show what the answer is based on, in a form the person can check in seconds. For an image, the basis is the area the AI reacted to. For an estimate, the basis is the past cases the estimate resembles. For a written answer, the basis is the passage in the source document.

Many AI tools hand a person a bare answer: a score, a warning, an estimate, a recommended price. The person then has to accept or reject the answer with nothing to go on.

What the evidence shows

  • A bare score made experienced inspectors worse. In an experiment at a Siemens factory, 48 workers checked circuit boards with an AI aid. Those who saw only the AI’s score were 88.6% accurate. The AI alone was 95.6% accurate. The score-only workers were 3.6 times more likely to overrule the AI when it was right.
  • The same AI with a picture made them better. Workers who also saw where the AI had found a problem were 96.3% accurate. They caught 93.0% of defective boards, against 82.0% for the score-only group. They overruled 96.9% of the AI’s wrong answers. The case page has the details.
  • It held for a second group. In the same study, 113 radiologists reading chest X-rays were 83.8% accurate with the picture and 79.1% with the score alone.
  • Similar past jobs beside each estimate. A steel fabricator built an AI tool that estimates production times for quotes. With each estimate it shows the ten most similar past jobs and their measured times. Staff can see at once if the estimate looks wrong and change it.
  • An answer engineers could act on. At a chip plant, researchers used a method that shows how much each production step contributed to the AI model’s prediction. It named steps and machines. Production experts then found the physical causes. A test of the two changes the model pointed to cut the loss of chips by 21.7%.
  • Marked areas in a screening trial. In a Swedish breast-screening trial with 105,934 women, the AI marked suspicious areas on each scan for the radiologists. It also decided which scans one radiologist read and which two read. The AI-supported group found 6.4 cancers per 1,000 women, against 5.0 with standard reading by two radiologists, with no rise in false alarms and 44.2% less reading work. The trial tested the whole set-up, so it does not show what the marked areas added on their own.

One finding goes the other way. A study that combined 106 experiments checked whether giving people an explanation of the AI’s output changed how well the person and the AI did together. On average the explanation made no difference. So the kind of explanation matters, and so does the task. Test it on your own work, as step 5 describes.

How to do it

  1. List where staff act on a bare answer. Go through your AI tools and note each place where a person sees a score, a warning, an estimate or a recommendation with no reason beside it.
  2. Decide what the basis should look like. Match it to the task.
    • Images: the area the AI reacted to, marked on the image. This is what the factory workers saw.
    • Estimates of time or cost: the most similar past cases, with what really happened. The steel fabricator shows ten.
    • Analysis of a process: the inputs that counted most, named in the terms your staff use. The chip plant’s engineers got steps and machines.
    • Written answers: the passage in the source document, so that the reader can open it.
  3. Ask to see what your staff’s screen will show before you buy. Have the vendor show what your staff will look at for one answer. If the screen shows a number alone, ask what else the tool can show. See ask AI vendors for proof.
  4. Tell staff the AI can be wrong. The factory workers were told so before they began. Say that the basis is there to be checked.
  5. Test both displays on your own task. Take a set of real examples where you know the right answer. Give half your staff the bare answer and half the answer with its basis. Compare both groups with the AI alone. The factory did this with 48 people and 200 images in ten days.
  6. Let staff change the answer. The steel fabricator’s staff can adjust any estimate. Record when they do, and whether they were right.
  7. Look up what the AI cites. A source the AI names is only a basis if it exists. See keep a person checking.

What to measure

  • The share of the AI’s wrong answers that staff overrule.
  • The share of its right answers that staff overrule.
  • The accuracy of staff with the tool, against the AI alone, on examples where you know the right answer.

Where it fails

  • It does not always help people catch errors. In the study above, radiologists who saw the picture overruled 50.8% of the AI’s wrong answers. Radiologists with the score alone overruled 57.7%. Their gain came from fewer false alarms.
  • The basis is made up. AI tools that write text can name sources that do not exist. A law firm’s own AI tool cited nine cases, and eight did not exist. One answer is to forbid the task. When England’s Solicitors Regulation Authority authorised Garfield.Law, a law firm whose services are delivered by AI, in May 2025, one of its conditions was that the AI may not propose case law.
  • The basis is unreadable. A marked image or ten similar jobs can be checked at a glance. A table of technical values cannot. Show it in the terms the person works in.
  • Nobody looks. If staff have no time to check, the display changes nothing. Allow the seconds it takes.

By company size

  • Micro and small: you will mostly buy tools. Prefer the one that shows its basis, and use step 3 when you compare vendors.
  • Medium: run the test in step 5 with a few dozen examples before you settle the process.
  • Mid-market and enterprise: write the requirement into purchasing and into anything built in-house, and track the two overrule measures for each tool.

Company cases

Companies that did this

Sources

  1. Senoner, Schallmoser, Kratzwald, Feuerriegel and Netland (2024). Explainable AI improves task performance in human–AI collaboration. Scientific Reports, 14, 31150. What this study did and found
  2. Vaccaro, Almaatouq and Malone (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293–2303. Limits or contradicts this practice. What this study did and found
  3. Hernström, Josefsson, Lång and others (2025). Screening performance and characteristics of breast cancer detected in the MASAI trial. The Lancet Digital Health. doi.org
  4. Solicitors Regulation Authority (May 2025). Authorisation of Garfield.Law, a law firm whose services are delivered by AI. sra.org.uk