When customer-facing AI goes wrong: five real cases and the control each one lacked

In November 2022, after his grandmother died, Jake Moffatt asked the chatbot on Air Canada's website about bereavement fares. It told him he could apply for the reduced rate within 90 days of the ticket being issued. Air Canada's own policy page, which the chatbot linked to, said the policy did not apply after travel. Air Canada refused the refund, and in February 2023 a representative admitted the chatbot had given “misleading words”.

In its decision of 14 February 2024, the British Columbia Civil Resolution Tribunal rejected Air Canada's suggestion that the chatbot was effectively a separate legal entity responsible for its own actions, calling it “a remarkable submission”. The tribunal found negligent misrepresentation: Air Canada had not taken reasonable care to ensure its chatbot was accurate. It awarded $812.02 in damages, interest and fees.

The tribunal recorded that Air Canada gave no information about how the chatbot worked, so it should not be assumed to be a large language model. The lesson holds either way.

The missing control: grounding. A customer-facing assistant should answer policy questions from the business's own current policy, and say so, or hand over to a person. If the bot and the policy page disagree, the business is answerable for both.

In January 2024, a customer trying to trace a missing parcel got DPD's chatbot to swear, write a poem criticising the company and call DPD “the worst delivery firm in the world”. The BBC reported that one post about it was viewed 800,000 times in 24 hours.

DPD said it had operated an AI element within the chat successfully for a number of years, that an error occurred after a system update the day before, and that the AI element was immediately disabled while it was updated.

The missing controls: regression testing after every change, including hostile and off-topic prompts, and limits on tone and scope. A bot that cannot help with the actual question should pass the customer to a person rather than improvise.

In December 2023, users discovered that the ChatGPT-powered chatbot on the website of Chevrolet of Watsonville, a California dealership, could be talked into almost anything. One user posted that he had bought a 2024 Chevy Tahoe for $1 after the bot “agreed”. Business Insider noted the exchange was not legally binding.

The chatbot's supplier, Fullpath, shut the bot down on that dealer's site. Its chief executive said that in the logs people “were at it for hours”, that most attempts failed, and that no confidential dealership data was disclosed.

The missing control: a hard limit on what the assistant may commit to. Prices, discounts, refunds and anything that looks like an offer should come from the business's systems or a person, never from free text generated in the moment.

In March 2024, The Markup, with Documented and THE CITY, reported that New York City's MyCity chatbot for business owners, built on Microsoft's Azure AI services, gave answers that would break the law: that employers could take a cut of workers' tips, that businesses could refuse cash despite a 2020 city law, and that landlords did not have to accept housing vouchers. Its answers were also inconsistent between users. A disclaimer warned it might occasionally produce incorrect, harmful or biased content.

The city kept the pilot running with added disclaimers and narrower answers. In January 2026, the new mayor, Zohran Mamdani, called it “functionally unusable”, and on 4 February 2026 the city took it down, saying its beta test had ended.

The missing controls: answers drawn only from authoritative sources and linked to them, testing against known questions with known correct answers before launch, and ongoing review of what the bot actually says. A warning that the bot may be wrong moves the risk to the user. It does not reduce it.

This case is about data protection rather than wrong answers. The UK Information Commissioner's Office investigated Snap's My AI chatbot, and on 21 May 2024 announced it was satisfied that Snap had now completed a risk assessment that complied with data protection law. No fine was issued. The ICO called the investigation “a warning shot for industry”.

The missing control: a data protection impact assessment before a customer-facing AI goes live, not after a regulator asks for one.

Read together, these cases are not about rogue technology. They are about ordinary controls left out: the assistant was not tied to the business's own rules, could promise what it should not, was changed without testing, had no clean route to a person, or launched before anyone assessed the risk.

Those are the controls we build into every workflow and check in an AI Control Review: answers grounded in your own sources, no commitments without approval, named test cases including the hostile ones, a human handover that works, a record of every action, and a risk assessment before launch.