AI-first company

Assessment · Interview guides and checklists

NASSS-CAT: complexity assessment tools for technology projects

What makes this technology project hard?

By
Last reviewed
Corrections
Report an error
Who answers The project team, and staff in interviews
How long Two checklists, a 35-question form and an interview guide
Can you use it? Free to use, ask before changing
Testing Moderate

Tested properly, but by one team or in one setting. Nobody independent has yet repeated the test.

In short

Two checklists, a monitoring questionnaire and an interview guide for finding what makes a technology project hard. Use them to plan an AI rollout with the people it affects.

For whoever runs the assessment: an HR or operations lead, an internal project team, or an outside consultant.

What it measures

The name stands for the ways a technology project can fail: non-adoption, abandonment, and failure to grow beyond the first site, spread or last. The four parts ask how complex a project is in several areas, among them the technology, the value it offers, the people who must adopt it, the organization and the wider setting.

They are:

  • SHORT: a quick checklist for an early conversation.
  • LONG: a detailed checklist with space for an action plan.
  • PROJECT: a 35-question form for tracking a project over time.
  • INTERVIEW: prompts for semi-structured interviews.

The idea behind them is simple. Projects that are complex in many areas at once rarely become routine. Find the complexity early and reduce it, or plan for it.

How to use it

  1. Start with SHORT in a meeting of the project team.
  2. Use LONG for the areas that came out as difficult.
  3. Use INTERVIEW with staff from different levels and jobs, managers included.
  4. Track the project with PROJECT.

They come from healthcare. One area asks about the medical condition being treated. In a company, replace it with the business problem the AI is meant to solve, or leave it out.

What it does not measure

  • Who decides what, between people and AI. A review of frameworks for AI found that this one does not cover dependence on data, human oversight or shared decisions.
  • What people do day to day. It does not observe behaviour.
  • A readiness score. The output is a list of difficulties and actions.

How it was tested

  • The framework was first applied to six technology programmes in 20 organizations, followed for up to three years. Programmes that were complex in many areas did not move into routine use.
  • A review found 57 studies that used the framework. Barriers clustered in two areas: the organization, and the people expected to adopt the technology.
  • It has been used to study AI in hospitals. In one study of 29 interviews, culture, leadership, trust between management and teams, and local champions helped. Conflicting views of what the AI was worth held it back.
  • The four parts were designed in workshops with 50 people involved in such projects and tried out in real projects.

There are no scores to test here in the way a questionnaire is tested. The rating reflects wide use of the framework and lighter evidence for the four parts themselves.

Licence and access

All four parts are free downloads without a formal licence. The authors’ guide says researchers should modify the interview prompts as appropriate and acknowledge the source. For the other three, ask the lead author before you publish a translation or an adapted version.

Alternatives

Other assessments with a similar job, and when to prefer each.

Practices that use it

The details

Testing
Moderate The framework behind them has been used in at least 57 studies, including AI projects in hospitals. The checklists, questionnaire and guide themselves were designed with 50 people involved in such projects and tried out in real projects. No results show that using them improves outcomes.
Looks at
Organization
How the answers are collected
Workshops, checklists filled in by a project team, and interviews.
Length
A short and a long checklist, a 35-question monitoring questionnaire, and a set of interview prompts
Cost
Free
Licence
No formal licenceAll four parts are free to download. The interview guide may be modified if you credit the source. For the checklists and the monitoring questionnaire the terms are not established: ask the lead author before you publish a translation.
Languages
English
By
Greenhalgh and colleagues, 2020

Sources

  1. Greenhalgh, Maylor, Shaw, Wherton, Papoutsi and colleagues (2020). The NASSS-CAT Tools for Understanding, Guiding, Monitoring, and Researching Technology Implementation Projects in Health and Social Care: Protocol for an Evaluation Study in Real-World Settings. JMIR Research Protocols. doi:10.2196/16861
  2. Greenhalgh, Wherton, Papoutsi, Lynch, Hughes and colleagues (2018). Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC Medicine. doi:10.1186/s12916-018-1050-6
  3. Shin, Hamovitch, Gatov, MacKinnon and colleagues (2025). The NASSS framework use over time: A scoping review. PLOS Digital Health. doi:10.1371/journal.pdig.0000418
  4. Alami, Lehoux, Papoutsi, Shaw, Fleet and colleagues (2024). Understanding the integration of artificial intelligence in healthcare organisations and systems through the NASSS framework. BMC Health Services Research. doi:10.1186/s12913-024-11112-x
  5. Gama, Tyskbo, Nygren, Barlow, Reed and Svedberg (2022). Implementation Frameworks for Artificial Intelligence Translation Into Health Care Practice: Scoping Review. Journal of Medical Internet Research. doi:10.2196/32215

Related studies on this site