How we rate evidence

In short

• A rating says how well a claim has been tested. It does not say how big the benefit is.
• There are four levels, from Strong to Common practice.
• Two flags warn you when studies disagree or when results depend on conditions.
• Company announcements are not evidence. Neither are worked examples.
• Every claim links to its source, so you can check it yourself.

The four levels

Two terms first. A rigorous study compares people or teams who used a practice with similar ones who did not. We count two kinds. In a controlled trial, researchers decide who gets the new tool or method. In a natural experiment, a change reaches some groups before others, and researchers compare them.

One example of a rigorous study: a study of real work at one company, covering more than 5,000 customer-support agents, found that agents with an AI assistant resolved about 14% more issues per hour (Brynjolfsson, Li and Raymond).

1. Strong

   Several rigorous studies, run by separate teams, reach the same result. They were done in conditions close to normal business.

   Example. Three research teams each run a controlled trial inside working companies. All three find the same effect.

2. Moderate

   One rigorous study. Or several studies or large surveys that agree with each other and explain their methods clearly.

   Example. One large field study in a real workplace, with a clear comparison between staff who had an AI tool and staff who did not.

3. Emerging

   Several documented company cases report similar results, with numbers. Nobody compared them with a group that did not use the practice.

   Example. Five companies publish before-and-after figures for the same change. None of them had a comparison group.

4. Common practice

   Widely recommended by people who do this work. No published study has tested it yet.

   Example. Advice that many experienced managers repeat, but that no published study has tested.

Common practice does not mean weak advice. It means experienced managers widely recommend it, and nobody has yet tested it in a study and published the result.

Assessments, such as questionnaires and knowledge tests, use the same four levels. There the lowest level is shown as “Not yet tested”: nobody has published a test of whether the assessment measures what it says it measures. What the levels mean for an assessment.

Two flags

A flag sits next to the rating. It tells you to read the detail before you act.

• Contested

  Studies disagree. We show both sides and do not pick one.

  Example. One study finds a clear gain. Another of similar quality finds none.

• Context-dependent

  The practice helps in some conditions and harms in others. The page says which.

  Example. AI help improves results on tasks the AI handles well, and makes them worse on tasks it handles badly.

What a rating does not tell you

• How big the gain is. A Strong rating means the finding is well tested. The gain itself may be small.
• Whether it fits your company. A study done in large firms may not hold in a firm of 20 people. Each page says where the evidence comes from.
• What it costs. Cost and effort are described separately from the rating.
• How useful it is. A practice with one dot can be the right first step. The dots only say how much testing stands behind it.

Priority and effort

Each practice also shows when to take it up and how much effort it needs. These two labels are our editorial judgement. They are not evidence, and they do not change a rating.

• Do first. Before you start with AI, or as you start.
• Do next. Once your first uses of AI are running.
• Do later. When AI is an established part of the work.

• Effort: low. Days, one person.
• Effort: medium. Weeks, a few people.
• Effort: high. Months, several teams.

We set both for a typical company, from the steps on the practice page. Your own order may differ. Each practice page says what changes with company size.

What does not count as evidence

Company memos and announcements are signals

A memo tells you what a company intends to do. It does not tell you whether it worked. In April 2025, for example, Shopify’s chief executive told teams to show why AI could not do a job before asking for more staff (the memo). That is a statement of intent. It is not a result.

We record announcements like this as signals. A signal never supports a rating.

Worked examples are illustrations

A worked example shows how a method is applied, step by step. It may use a fictional company. It helps you picture the work. It does not show that the method produces results, so it carries no rating and never supports one. A practice may link to one to show what it looks like in use. See the worked examples.

Figures a company reports about itself are labelled

Some figures come only from the company they describe, or from the vendor that sold the tool. We say so wherever such a figure appears.

The same goes for company cases. A case is a named company with numbers and a public source. It carries no rating. Cases that companies report about themselves can support an Emerging rating at most, and only when several agree.

How we decide a rating

1. We collect the published sources on a practice or assessment.
2. We read each source in full. A rating never rests on a source we have not read.
3. We note how each study was done, who took part and where its results stop applying.
4. We match the sources against the four levels above and add a flag if one applies.
5. We link every claim to its source.

How often we review

• Every page shows the date it was last reviewed.
• We review each rated page at least once every 12 months.
• We review a page sooner when an important new study appears.
• A rating can go up or down after a review.

Version and changes

This is version 1.1 of the method. Each change to it is listed here.

Version 1.1 · 13 October 2026

• The lowest level is now called “Common practice”. It was “Practitioner view”. What counts for each level has not changed.
• On assessments the lowest level is shown as “Not yet tested”. The level and what counts for it are the same.

Version 1.0 · 13 October 2026

• First published version.
• Four evidence levels and two flags.
• Rules for company announcements and for worked examples.
• Review interval set at 12 months.