At a glance
- Company
- AshurstInternational law firm
- Size
- Enterprise (1,000+ employees) The trials involved 411 people across 23 offices in 14 countries.
- Industry
- Professional services
- Where and when
- United Kingdom and 13 other countries. November 2023 – March 2024, published June 2024.
- Who produced the figures
- The company itselfThe figures are the firm’s own, from the report it published on its trials. The timed experiments used small groups, and the blind test covered four court cases.
What they did
From November 2023 to March 2024 the firm’s digital team ran three trials of generative AI tools. They involved 411 partners, lawyers and support staff from every practice area, in 23 offices.
The team wrote down six assumptions before it began and set out to test them. It gathered evidence in three ways:
- Timed experiments. Small groups did a real legal task with and without the tools. About a quarter of participants took part in at least one.
- A blind test. Lawyers and AI tools each summarised four court cases. Four senior lawyers scored the summaries without knowing who had written them, and guessed which were AI.
- Surveys before, during and after.
A group that included the firm’s general counsel and its risk and security teams set rules for each trial. Only public data was used.
The numbers
Time saved on a first draft, in the timed experiments:
- About 80% on UK company filings that draw on a company’s articles of association.
- About 59% on research reports that draw on public company filings.
- About 45% on first-draft legal briefings.
- Up to two hours a task on non-legal writing such as job descriptions, policies and social media posts.
Accuracy, in the blind test:
- Lawyers scored higher on average: 3.5 out of 5, against 3 out of 5 for AI drafts.
- AI drafts varied more. They scored from 1 to 4. Lawyers’ drafts scored 3 or 4.
- In 67% of cases the AI draft was at least as accurate as the lawyer’s first draft.
- Half the AI drafts were taken for human work, or the experts could not tell. Every lawyer’s draft was recognized as human.
- One AI summary cited a wrong verdict. The expert who saw it gave the whole piece the lowest score on every measure.
From the surveys:
- 77% agreed that the tools got them to a first draft quicker.
- 67% still recommended adopting the tool they had tried, though they had seen it make things up.
- 88% in the largest trial said the experience made them feel more prepared for the future.
What the firm concluded
- The value is in first drafts, and in gathering the source material for them. People spoke of no longer starting from a blank page.
- A saving holds only if the draft is right. A wrong first steer can cost more time than it saves once the work is checked.
- Results differ by task. Collecting lists of uses was not enough. People had to try the tools on their own work.
- Easy to use is not the same as useful. Staff needed support and a place to compare notes. Short trials with deadlines and friendly competition between offices kept people engaged.
What the case shows
- Measure speed and accuracy separately. The tools were much faster and, on average, a little less accurate. Both facts matter to the decision.
- A trial can be set up to answer questions. The firm wrote its assumptions down first and chose its tests to match.
- Checking stays. Made-up content appeared in every tool tested. A reviewer who knows the subject is part of the cost.
- No client data was needed to learn this. The whole trial ran on public material.
What this says about the practices
Practices the company applied
- Measure time saved per task against a baseline The firm timed set tasks under controlled conditions. It also had senior lawyers score AI drafts and lawyers’ drafts without knowing which was which. Do first · Effort: low (days, one person)
- For each kind of task, test whether a person with AI beats either one alone The firm tried the tools on as many kinds of task as it could. It found the results uneven from one task to the next. Do next · Effort: medium (weeks, a few people)
- Decide what data may go into which AI tools The firm’s general counsel, risk and information-security teams set strict rules for each trial. Only public data was used, and no client data. Do first · Effort: low (days, one person)
- Keep a person checking anything that goes to a customer or carries legal weight Participants reported frequent made-up content from every tool. The firm concluded that AI work must be “thoroughly and properly reviewed by appropriate experts”. Do first · Effort: medium (weeks, a few people)
Sources
- Ashurst (10 June 2024). Vox PopulAI: Lessons from a global law firm’s exploration of generative AI. Report. ashurstperkinscoie.com
- Dell’Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon and Lakhani (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science, 37(2), 403–423. What this study did and found