Target's pregnancy prediction and data-poisoning fragility
Addie (15 years in cybersecurity) opens with a real-world example: Target's data team built a statistical model that produced a pregnancy prediction score by spotting 'around two dozen product combos' — not baby items but small shifts like 'unscented lotion, vitamin and magnesium supplements, and bigger purses' — and timed coupons to capture long-term brand loyalty. From that concrete case she generalizes to a class of failures called data poisoning: slipping corrupted examples into training data so models learn confident, wrong patterns. At a human level this can be playful trolling (binge-watching irrelevant videos) or malicious insertion of poisoned records.
She lists four core assumptions continuous-learning systems rely on:
- 'First, your clicks reflect real interest.'
- 'Next, watch time reflects real attention.'
- 'Third, purchases reflect real intent.'
- 'Fourth, your past behavior predicts your future behavior too.'
Two technical demonstrations show why scale helps attackers: researchers found that in a '400 million image' dataset poisoning about '0.01%' (tens of thousands of images) — costing 'around $60' to host/upload — could backdoor model behavior while leaving benchmark accuracy intact. Another team and 'Anthropic' showed inserting roughly '250 carefully written documents' into a corpus of billions can reliably backdoor language models. The core mechanism: plausible, repeated signals in the right context get absorbed and are invisible to manual auditing.
Practical consequences: digital advertising, built on probabilistic identity and claims like a '0.86 chance' of purchase intent, breaks down when clicks are faked or bots flood signals, wasting budgets and eroding trust. At web scale the same poisoning causes 'model collapse' as AI systems retrain on recycled AI-written sludge.
Defensive options: Addie presents two responses. Collective resistance: intentionally flood targeting systems with plausible contradictory signals to dissolve confidence in harmful risk scores. Individual action: create a new social account and deliberately refuse to feed rage bait; engage only with content that improves life skills, joy, health and calm. Over time the recommender will relearn and the user's feed becomes a tool, not a trap.