Privacy · the failure cases
When AI leaks
30-second gist~30s read
AI services have leaked user data in ways nobody expected: chat titles surfacing in other people's accounts, training data containing real personal details, internal tools accidentally exposing secrets pasted into prompts.
Big providers fix these quickly. The lesson isn't to flee — it's to be careful what you paste in.
If you want more
Three real cases
- The ChatGPT sidebar bug, March 2023. A caching error briefly let some logged-in users see other users' chat titles in their history sidebar. Later disclosure showed the same incident also exposed limited account data — names, emails, partial payment information — for a small percentage of subscribers. OpenAI patched it within hours and disclosed publicly. Around the same time, Italy's data regulator temporarily banned ChatGPT, citing broader GDPR concerns about training-data use and age verification — separately from the sidebar bug.
- The Samsung paste, March 2023 (publicly reported April 2023). Engineers at Samsung pasted internal source code into ChatGPT to debug it. The code became part of OpenAI's data. Samsung banned employee use of public AI tools and rebuilt internal alternatives. The episode is now used as the canonical "do not paste confidential work into a public AI" lesson.
- Training-data extraction, 2023-2024. Researchers showed that asking some models to "repeat this word forever" or carefully phrased prompts could cause them to dump fragments of their training data — sometimes including real names, phone numbers, and email addresses. Patched, but a reminder that "training data" isn't fully forgotten.
What this means for you
For the public free tools, treat your prompts roughly the way you'd treat a well-managed third-party service. Reasonable for most things; not appropriate for confidential work, secrets, banking details, or anything you'd be sad to see in a leak. For genuinely sensitive work, use your organisation's enterprise AI or run an open-weights model locally.