AI-first company

People and skills · Adopt

Test AI knowledge instead of asking staff to rate their own

People judge their own skills poorly. To find out what your staff know about AI, use a short knowledge test or a second opinion, not only a self-rating.

By
Last reviewed
Corrections
Report an error
When Do next. Once your first uses of AI are running.
Effort Medium (weeks, a few people)
Evidence Moderate

A review of 16 AI-literacy questionnaires and a study of 629 people agree that self-ratings are a weak measure. The 629 were students, and the skills were digital skills in general. No study has compared the two methods inside companies using AI.

Fits
Industries All industries
In the guide Mid-market companies

What it is

Add a check that does not depend on self-rating: a short knowledge test, a supervisor’s view, or both. Most training plans start with a survey that asks staff how well they understand AI. The answers measure confidence more than knowledge.

What the evidence shows

  • Most questionnaires are self-ratings. A review of 16 AI-literacy questionnaires found that 13 ask people to rate themselves. Only 3 test what people can do. The review searched for all published studies on the question, by fixed rules.
  • Self-ratings and tests disagree. In a study of 629 students, self-rated digital skills differed from test results beyond chance. Some people were overconfident and some underconfident.
  • Supervisors see something different. At one financial company, employees and their supervisors gave different accounts of the employees’ digital skills.
  • Real knowledge changes results. In an experiment, people who had been taught how AI works chose better which tasks to hand to AI, and performed better.

Knowledge tests for AI now exist with open licences. Two are SAIL4ALL and AICOS.

How to do it

  1. Decide what staff need to know for their role: what AI can and cannot do, how to check its output, what data must not go into it.
  2. Use a short knowledge test with right and wrong answers. Keep it to 15 minutes.
  3. Add a self-rating if you want one. The gap between the two is useful: it shows who is overconfident.
  4. Ask supervisors about the same skills, for a third view.
  5. Make the test low-stakes. Use the result to plan training, never to rank people.
  6. Test again after training.

What to measure

  • Test scores by team, before and after training.
  • The gap between self-rating and test score.

Where it fails

  • A test people fear. If staff think the score affects their job, they will resist it or cheat. See protect psychological safety.
  • Knowing is not doing. A knowledge test does not show whether someone uses AI well in daily work. Look at work samples too.
  • Translated tests. None of the 16 questionnaires had been tested in other countries or cultures. A translated test may not measure quite the same thing.
  • Regulation. In the European Union, the AI Act asks firms that use AI to support their staff’s AI literacy. A test result is one way to show where you stand. The Act does not require a test.

By company size

  • Medium: one short test for everyone, plus a conversation with each team lead.
  • Mid-market and enterprise: set what each role needs to know, and test against that.

Assessments for this practice

These are existing questionnaires, tests and guides you can use. Each card gives the question it answers, who answers it and whether you can use it freely. All assessments.

Company cases

What happened without it

Sources

  1. Lintner (2024). A systematic review of AI literacy scales. npj Science of Learning, 9, 50. What this study did and found
  2. Luna, Ocaña, Ortiz-Colón and Rodríguez Moreno (2026). From Self-Perception to Reality in Digital Competence: Findings from the DigComp 2.2 Framework. The Electronic Journal of e-Learning, 24(3), 137–153. What this study did and found
  3. Rogacka (2022). Digital skills in the workplace – from the employees’ and supervisors’ perspective. e-mentor, 5(97), 53–64. What this study did and found
  4. Pinski, Adam and Benlian (2023). AI Knowledge: Improving AI Delegation through Human Enablement. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. What this study did and found