Millions of people now ask a chatbot before they ask a doctor. The answers arrive instantly, sound confident and cost nothing. But a new analysis in Nature Health argues that the deepest problem with AI-generated health information is not simply that chatbots sometimes get things wrong — it is that when they do, the error usually leaves no trace at all.
The paper, titled “The invisible risks of AI-generated health information”, lays out why harm from medical advice given in a private chat window is so hard to detect, measure or correct — and why health systems currently have almost no machinery for catching it.
A consultation nobody can audit
When a medicine harms a patient, there is a reporting system for that. Pharmacovigilance networks exist precisely so that adverse reactions are logged, patterns are spotted and dangerous products are withdrawn. A chatbot conversation has no equivalent. The exchange is private, often deleted, and invisible to regulators, researchers and even the companies running the models once the session ends.
That invisibility cuts both ways. A user who follows bad advice and gets worse may never connect the outcome to the chat — and even if they do, there is no standard way to report it. Multiply that across hundreds of millions of health queries, and even a small error rate could translate into real-world harm that never appears in any dataset.
Confident, fluent — and sometimes wrong
The risks are not hypothetical. The analysis points to documented cases in which AI-influenced decisions contributed to serious harm, including a published clinical case of bromism — bromide poisoning — influenced by AI use, and litigation in the United States alleging that chatbot health advice contributed to a near-fatal crisis. Research cited in the paper also shows that large language models can repeat medical misinformation persuasively, that repetition itself makes false claims more believable, and that models are vulnerable to poisoned data.
Part of the problem is presentation. Chatbots answer in fluent, authoritative prose whether they are right or wrong, and they rarely signal uncertainty in a way lay readers can judge. Studies of how people actually use these tools show health is one of the most common reasons for turning to a general-purpose chatbot — often for symptoms, diagnoses and treatment questions that would once have gone to a clinician or a regulated symptom-checker.
Researchers are flying blind too
Compounding the problem, independent researchers have lost much of their visibility into how these systems behave at scale. As platforms have restricted data access in what academics call the “post-API age”, outside scientists struggle to study how health misinformation spreads through AI answers, or even how often dangerous answers occur. The paper argues this creates a accountability gap: the companies deploying the models hold nearly all the data, while the public bears nearly all the risk.
It is a pattern our coverage has seen elsewhere in AI safety: capabilities are moving faster than the oversight around them, whether the question is AI models learning to copy themselves onto everyday devices or health tools being marketed at scale, as in the digital marketing strategies now targeting healthcare organisations.
What the authors want instead
The paper’s authors argue for treating AI health information more like a regulated product: standardised ways to report suspected chatbot-related harm, better audit access for independent researchers, clearer uncertainty labelling, and agreed safety standards for general health queries — an area where consensus roadmaps are beginning to emerge. They also note that courts are starting to test liability, including a German ruling holding a search provider responsible for an AI-generated falsehood.
Until then, their advice to readers is unglamorous but sound: treat a chatbot’s medical answer as a starting point for questions, not as a diagnosis; check anything consequential against a clinician or a recognised health authority; and be most sceptical of the answers that sound the most certain.