To judge an AI incident report, ask who is speaking, whether anyone else confirms it, whether the evidence can be checked, and what the account does not say. Prefer court records, vendor bulletins and named reporting over screenshots and secondhand summaries.
Why care
Stories about AI going wrong travel quickly, and some are wrong. When Google explained odd AI Overviews answers in May 2024, it said many screenshots on social media were faked, and that examples about leaving dogs in cars never appeared in Search. At the same time, it confirmed real odd answers, such as the one on how many rocks to eat. Both facts were true together.
If you cite incidents in training, policy or sales, a fake one will cost you credibility.
Six questions
1. Who is the source? A vendor bulletin, a court order, a named reporter, a researcher with a write up, or an anonymous post. Each carries different weight.
2. Is it one sided? The July 2025 Replit and SaaStr account relied mainly on the customer's posts. The Register said it sought comment and had no official statement at publication. That does not make the account false. It means you should say "reported by."
3. Can the evidence be checked? A tribunal decision, a security bulletin with a version number and hash, or a published paper can be read directly. A cropped screenshot cannot.
4. Are the numbers consistent? Summaries of the same decision can list different totals. When figures differ, find the primary document or leave the number out.
5. What is missing? The Samsung reports said the memo did not say what the code was. A good report names its gaps. A bad one fills them with guesses.
6. Is there a correction trail? Serious sources update their pieces. The AWS bulletin on the Amazon Q extension carries an update date.
A quick ranking
Strongest to weakest, as a rule of thumb:
- Court or tribunal records
- Vendor security bulletins and postmortems
- Peer reviewed papers and conference talks
- Named journalism with documents
- Researcher blog posts without vendor reply
- Social posts and screenshots
This is a heuristic. A careful blog post can beat a sloppy article.
How to write about an incident
- Say who reported it.
- Use words like "reported" and "said" for claims you cannot confirm.
- Avoid saying a system "decided" or "lied."
- Leave out any number you cannot trace.
- Link the source.
See a blameless review template.
Quick check
Are screenshots useless?
No, but they are weak alone. Look for the original post, the date, and a second source.
What if a vendor never responds?
Say so, as The Register did, and describe the account as reported by the customer.
Sources
- Google: AI Overviews, about last week, 30 May 2024
- The Register: Replit and SaaStr incident report, 21 July 2025
- Mondaq: Airline ordered to compensate a BC man because its chatbot provided inaccurate information
- CSO Online: Samsung bans staff AI use over data leak concerns
- AWS security bulletin AWS-2025-015
Last reviewed: 19 October 2026.
