When It Goes Wrong: AI Incidents and What They Tell Us
Understanding AI risk in the abstract is one thing. Seeing it play out in a real situation is another. In this second part of the series, we look at five scenarios that illustrate how AI-related incidents actually unfold – what triggers them, how they escalate, and what they reveal about the governance gaps that made them possible.
These scenarios are illustrative rather than drawn from specific cases, but each reflects patterns we encounter regularly in how organizations are using AI today. As you read them, consider which feel closest to your own organization’s situation. That discomfort is useful information.
Scenario 1: The confident wrong answer
A professional services firm introduces an AI assistant to help staff research regulatory requirements for client engagements. The tool is fast and articulate. Staff quickly come to rely on it. On one engagement, the AI produces a detailed and plausible summary of a regulatory position that is factually incorrect – the regulation it references was amended eighteen months ago. The error is not caught before it reaches the client. The client acts on the advice. The consequences are significant, and the firm faces a professional negligence claim.
The governance question this raises: Is there a human review step before AI-generated outputs are used in client-facing work? Do staff understand that AI tools can be confidently wrong?
Scenario 2: The hiring tool that discriminated
A mid-sized company implements an AI-powered screening tool to manage a high volume of job applications. The tool scores candidates based on patterns in historical hiring data. Over time an audit reveals that the model consistently scores candidates from certain demographic groups lower – not because of any explicit instruction to do so, but because the historical data it learned from reflected past hiring biases. A regulatory investigation is opened. The company faces both reputational damage and potential legal liability, despite never having intended to discriminate.
The governance question this raises: Do you know how any AI tools used in employment decisions make their recommendations? Has bias testing been conducted? Is there a human in the loop before decisions are made?
Scenario 3: The data that should not have left the building
A finance team at a mid-sized business starts using a publicly available generative AI tool to help draft board reports and summarize financial data. It saves hours each week. Nobody has explicitly said they cannot use it. Over several months, significant volumes of commercially sensitive financial information, including unannounced performance data and strategic plans, are entered into the tool as prompts. The tool’s terms of service state that inputs may be used to improve the model. The information has, in effect, left the organization. The consequences surface during a due diligence process.
The governance question this raises: Does your organization have a clear policy on what information employees can and cannot share with external AI tools? Do employees know that public AI tools are not confidential environments?
Scenario 4: The vendor’s AI becomes your problem
A logistics company relies on a third-party supply chain management platform that has quietly introduced AI-driven demand forecasting into its core functionality. The AI makes a significant forecasting error during a critical period, leading to substantial stock shortfalls and missed client commitments. The logistics company faces contractual penalties and client claims. When they turn to the software vendor, the vendor’s terms of service disclaim all liability for AI-generated outputs. The exposure stays with the logistics company.
The governance question this raises: Do you know which of your software vendors have introduced AI into their products? Have you reviewed their terms of service for AI-related liability provisions? What contractual protections do you have?
Scenario 5: The AI that nobody was watching
A retail business implements an AI-powered customer pricing tool that dynamically adjusts prices based on demand signals, competitor data, and customer behavior patterns. The tool runs largely autonomously. Over time, the pricing patterns begin to look – to a regulator examining the sector – like coordinated pricing behavior. The company had no intent to engage in anti-competitive conduct. The AI did what it was optimized to do. The regulatory investigation does not distinguish between human and machine intent.
The governance question this raises: Do you have visibility into what your AI systems are optimizing for? Are there regular human reviews of AI-driven outputs in commercially sensitive areas? Do you have audit trails that demonstrate oversight?
What these scenarios have in common
None of these incidents required a dramatic technical failure. No system was hacked. No catastrophic outage occurred. Each one unfolded through a combination of normal business activity, reasonable-seeming decisions, and the absence of governance that would have caught the problem before it became a liability.
That is the nature of most AI-related risk in practice. It is not exotic. It is the gap between how fast AI has been adopted and how slowly governance has followed.
Each scenario also illustrates something important about where liability lands. In every case it lands on the organization – not the AI vendor, not the model, not the tool. The organization is the data controller, the employer, the service provider, the contracting party. AI does not change those legal relationships. It just creates new ways for things to go wrong inside them.
What insurers are watching
From an insurance perspective, these scenarios are not theoretical. Claims activity involving AI is increasing. Insurers are seeing it in professional indemnity, in cyber, in employment practices liability, and in directors and officers’ liability insurance. Underwriting guidelines are being fitted for AI risks and policy wordings are being updated. Underwriters are increasingly asking harder questions about AI governance than they were two years ago and there is no current indication that this wearing off anytime soon.
What that means practically is that how your organization manages AI risk is beginning to affect your insurance position – not just your operational risk. Organizations that can demonstrate governance maturity are accessing better terms. Organizations that cannot may find capacity restrictions or conditions attached.
What comes next
In Part 3, we move from understanding the risk to managing it – and deliver the practical framework and checklist that this series has been building toward.
About the Financial Lines Team
This series is written from within RiskPoint’s Financial Lines team, where we work with liability, governance, and emerging risk issues every day. The perspectives in this article are grounded in practical underwriting experience and close dialogue with brokers and clients navigating a changing risk landscape.
The Author
Erdal Erdogan is Technology Practice Lead at RiskPoint Group, where he focuses on insuring the tech industry across the Nordic and European specialty market. With over a decade of experience underwriting technology and software companies – including senior roles at large international insurance carriers – he specializes in Tech E&O/PI, Cyber, and from simple to complex technology risks. He works at the intersection of technology insight and insurance, helping brokers and clients navigate emerging and evolving risk landscapes.
About The RiskPoint Group
The RiskPoint Group is one of Europe’s largest Managing General Underwriters (MGUs), providing best in class insurance solutions to businesses and their advisors globally. The RiskPoint Group operates as RiskPoint in Europe and Asia and RP Underwriting in the rest of the world. With 325+ employees and 17 locations in Europe, North America, and Asia Pacific, the RiskPoint Group employs expert underwriting and specialized, in-house claims teams within Accident & Health, Transactional Risk, Property & Construction, Renewable Energy, and Liabilities, including Financial Lines, Cyber, Casualty, and Life Science. The RiskPoint Group is a Coverholder with Lloyd’s of London and is backed by a strong panel of well-reputed insurance companies.
Legal Notice & Disclaimer
The content published in this series – including all articles and accompanying materials – is produced for informational and educational purposes only. It does not constitute legal, regulatory, financial, or insurance advice, and should not be relied upon as such. The views expressed are those of the author in a personal professional capacity and do not necessarily represent the official position of RiskPoint Group or any of its affiliates, subsidiaries, or business partners.
References to the EU AI Act and other regulatory frameworks are provided for general informational purposes only. Regulatory obligations vary by jurisdiction, organization type, and the nature of AI systems deployed. Readers should seek independent legal advice regarding their specific compliance obligations.
Nothing in this content creates a client relationship, advisory relationship, or any other professional relationship between the author, RiskPoint Group, and the reader.
This content was accurate to the best of the author’s knowledge at the time of publication. AI regulation and market practices are evolving rapidly; readers should verify current requirements independently.
