What was missing most often
Each case page lists the practices that would have prevented the failure or made it smaller. These were missing most often across the 17 cases.
- Keep a person checking anything that goes to a customer or carries legal weight: missing in 8 of 17 cases
- Claim only what your AI has been tested to do: missing in 6 of 17 cases
- Ask AI vendors for proof before you buy, and test on your own material: missing in 5 of 17 cases
- Know the rules that already apply to your AI use: missing in 5 of 17 cases
- Budget for the work AI creates: missing in 4 of 17 cases
- When AI output goes wrong, correct it fast and in the open: missing in 3 of 17 cases
Made-up citations nobody checked
Lawyers and a consulting firm handed in work that cited court cases or sources that were wrong or did not exist. The case pages show who signed the work and what it cost them.
- Levidow, Levidow & Oberman: fined after filing court papers with cases ChatGPT made up A lawyer used ChatGPT for legal research and filed six court decisions that did not exist. The court fined the firm and two of its lawyers. The judge said using AI was not the problem. Not checking was.
- Morgan & Morgan: eight made-up cases from the firm’s own AI tool, signed by three lawyers A lawyer asked his firm’s own AI tool to add case law to a court filing. Eight of the nine cases it cited did not exist. Two other lawyers signed without reading. The firm had trained staff, and it then added a warning inside the tool.
- Butler Snow: a firm with an AI policy, and a partner who did not follow it The firm’s policy required written approval before using ChatGPT for legal research. A partner used it without approval and filed five citations it had made up. The court removed three lawyers from the case. The firm paid for a check of 2,400 citations.
- Primus Solicitors: 18 court citations that did not exist, taken from a client’s AI research A solicitor at a small law firm filed a statement with 45 case citations. Eighteen of the cases did not exist. His client had produced the research with public AI tools, and nobody at the firm checked it. The court referred him to his regulator.
- Deloitte Australia: a government report reissued to fix wrong citations A Deloitte report for an Australian government department had to be reissued to correct citations and a summary of a court case. The corrected version disclosed the use of a generative AI tool. The report does not say AI caused the errors.
AI claims that regulators challenged
These companies sold or advertised AI that a US regulator said did not do what they claimed. The case pages give what the regulator found and what the company paid or agreed to.
- DoNotPay: a “robot lawyer” that was never tested against a lawyer DoNotPay told customers its AI could do the work of a lawyer. The US consumer regulator said the company had never tested whether that was true and employed no lawyers to check. It paid $193,000 and had to tell past subscribers about the limits.
- accessiBe: an AI plug-in sold as making any website accessible, and a $1 million order accessiBe sold website owners an AI plug-in and said it would make any site meet accessibility standards. The US consumer regulator said it did not, and that reviews praising it were paid for. The company paid $1 million.
- Workado: an AI detector sold as 98% accurate that tested at 53% Workado sold a tool that claims to tell AI-written text from human writing, and advertised it as 98% accurate. The US consumer regulator said independent testing put it at 53% on general text. Buyers had only the vendor’s word.
- Delphia and Global Predictions: fined for saying they used AI that they did not have Two investment advisers advertised AI they did not have. One said its AI learned from client data. The other called itself the first regulated AI financial advisor. The US securities regulator fined them $225,000 and $175,000.
- Presto Automation: the US regulator found most “AI” drive-through orders needed a person Presto sold restaurants a voice AI for drive-through orders and told investors it eliminated human order-taking. The US securities regulator found that people working off site, mainly in the Philippines and India, entered the vast majority of orders. Presto settled without admitting or denying the findings.
- Air AI: the US regulator says small firms were sold “AI that replaces staff” and lost up to $250,000 Air AI sold small-business owners “conversational AI” that it said could replace customer-service staff, with promises that they would earn their money back. The US consumer regulator alleges that buyers lost up to $250,000 each and that promised refunds were rarely paid. Under a proposed settlement the owners are banned from marketing business opportunities.
AI that customers and shoppers dealt with
A chatbot gave a customer wrong advice. Facial recognition in shops scanned shoppers and wrongly flagged some of them. A tribunal or regulator found against each company, at least in part.
- Air Canada: held responsible for what its website chatbot told a customer Air Canada’s website chatbot gave a customer wrong information about refunds. A Canadian tribunal ordered the airline to pay. It rejected the argument that the company was not responsible for its chatbot.
- Rite Aid: banned from facial recognition for five years after buying it without testing it Rite Aid used facial recognition to spot suspected shoplifters. The US consumer regulator said the chain never asked its vendors how accurate the system was, never tested it and did not train staff. Shoppers were wrongly accused. The chain was banned from the technology for five years.
- Bunnings: facial recognition on every shopper in 63 stores, and what the regulator and the tribunal decided Bunnings scanned the face of every person who entered 63 of its stores to spot people it had flagged for crime or violence. The privacy regulator found that it broke the law. A tribunal later held that the chain could use the technology to fight serious retail crime, but agreed that it had not told shoppers properly and should have done a formal, documented risk assessment first.
Personal data typed into a public chatbot
Staff at two Australian government bodies put personal details of members of the public into ChatGPT. One body had not approved the tool. The other had no rules for it at all. Afterwards the first body blocked such uploads, and a regulator ordered the second to block the tools.
- NSW Reconstruction Authority: a temporary worker uploaded data on 2,031 people to ChatGPT A temporary staff member uploaded a spreadsheet of flood-recovery applicants to ChatGPT, which the agency had not approved. It held personal and health details of 2,031 people. The agency then wrote AI guidance, tightened training and blocked such uploads.
- Victoria’s child protection service: a worker put case details into ChatGPT, and the regulator ordered the tools blocked A child protection worker typed personal details of a family into ChatGPT to draft a court report. The report contained errors. The privacy regulator found the department had no rules, no training and no technical limits for such tools, and ordered it to block them.
A forecast the business bet on
Zillow bought homes at prices set partly by its own forecasts. When the forecasts missed, it closed the business. Zillow also named labour and supply problems.
For cases with measured or company-reported results, see all company cases. The labels and who produced each figure are explained in how the site rates evidence.