AI-first company

People and skills · Run

Check now and then what staff can do without the AI

On a sample of cases, have people do the work without the AI and compare the result with their usual work. Start with the people who check AI output and the people still learning the job, and set the tools that learners use to give hints, not answers.

By
Last reviewed
Corrections
Report an error
When Do later. When AI is an established part of the work.
Effort Medium (weeks, a few people)
Evidence Emerging

Two studies measured what people could do once the AI was taken away: one compared doctors’ results before and after AI arrived, and one was a randomised experiment with school students. Neither was in office work. No study has tested whether regular unaided checks prevent the loss.

Fits
Industries All industries

What it is

Pick a few cases every so often and have staff do them without the AI. Compare the result with what the same people achieve with it, and with what they achieved before the tool arrived. Start with two groups: the people whose job is to check what the AI produces, and the people still learning the job.

Most measures of AI at work look at results with the tool switched on. They cannot show whether people are losing the skill to do the work, or to spot when the AI is wrong. A check without the tool can.

What the evidence shows

  • Experienced doctors lost skill within three months. At four endoscopy centres in Poland, researchers compared colonoscopies done without AI before and after AI was introduced for other procedures. In the procedures done without AI, the share in which doctors found a precancerous growth fell from 28.4% to 22.4%. The study compared records from two periods. Other changes over the same months cannot be ruled out.
  • Practice with an unrestricted AI raised scores, and the exam without it lowered them. In a randomised experiment with nearly 1,000 secondary-school students in Turkey, an unrestricted version of GPT-4 raised practice scores by 48%. On the exam without AI, those students scored 17% below students who never had it. A version set to give hints, not answers, raised practice scores by 127%, and its students did about as well on the exam as those without AI.
  • The loss can go unnoticed for years. A case study of an accounting firm found that years of relying on automated software wore down staff skills. Nobody noticed, because the software kept working.
  • Learners lose the practice first. A study of robotic surgery found that the new technology sharply cut what trainees did themselves.

Neither of the two measured studies was in an office. Both measured what people could do once the AI was taken away, which workplace studies of AI almost never do.

How to do it

  1. Choose the tasks where the skill still matters. These are tasks where a person checks AI output, signs off the result, or must take over when the tool fails.
  2. Start with the checkers and the learners. The doctors in the study above were experienced. Checkers lose skill too, not only juniors.
  3. Take a small sample of real cases. A few cases a quarter per person is enough to see a trend.
  4. Have the person do them without the tool. Score the result against a known answer, or have a colleague score it without knowing whose it is.
  5. Compare like with like. Set the unaided result beside the same person’s results with the tool, and beside results from before the tool arrived if you have them.
  6. Set learner tools to give hints. Where your tool allows it, set it to explain and prompt, not to hand over the finished answer. In the school experiment, the hints-only version kept exam scores level.
  7. Act on a drop. If unaided results fall, give the person more work without the tool, or more coached practice.

What to measure

  • Results without the tool, per person, over time.
  • The gap between results with and without the tool.
  • How often checkers catch errors you have planted in AI output on purpose.

Where it fails

  • It costs time. Work done without the tool is slower. Keep the sample small.
  • Some skills are no longer needed. If nobody will ever do the task without the tool again, losing that skill may be a fair price. Keep the checks for skills people need in order to judge the AI’s output.
  • People may treat it as a test of them. Present it as a check on the tool and the training, not on the person.
  • The evidence comes from medicine and school. No study has yet measured this in office work.

By company size

  • Micro and small: do it informally. Once a quarter, do one real task the old way and see whether it still comes easily.
  • Medium and mid-market: build a few unaided cases into the review of each team that checks AI output.
  • Enterprise: make it part of quality control for each AI tool that a person signs off, with results tracked by team.

Sources

  1. Budzyń, Romańczyk, Kitala and colleagues (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology & Hepatology, 10(10), 896–903. What this study did and found
  2. Bastani, Bastani, Sungu, Ge, Kabakcı and Mariman (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. What this study did and found
  3. Rinta-Kahila, Penttinen, Salovaara, Soliman and Ruissalo (2023). The Vicious Circles of Skill Erosion: A Case Study of Cognitive Automation. Journal of the Association for Information Systems, 24(5), 1378–1412. What this study did and found
  4. Beane (2018). Shadow Learning: Building Robotic Surgical Skill When Approved Means Fail. Administrative Science Quarterly, 64(1), 87–123. What this study did and found