AI in Medicine: When Life-Saving Tools Become a Fatal Threat
In January 2026, OpenAI launched its product, ChatGPT Health, promising to assist users in interpreting medical reports and preparing for doctor appointments. However, by February of the same year, a study published in Nature Medicine revealed that the model failed to distinguish between emergency and non-emergency cases in more than half of the clinical scenarios tested, while also failing to detect suicidal ideation in multiple instances. This stark contradiction between marketing promises and real-world performance encapsulates a fundamental crisis: artificial intelligence is entering hospitals, doctors' clinics, and patients' lives at a pace far outstripping our ability to comprehend and regulate its attendant risks.
The question is no longer "Will AI transform medicine?", but rather "How do we prevent this transformation from turning into a silent catastrophe?" With the market value of AI-enabled medical devices surpassing $13.7 billion by 2025 and tens of thousands of such devices licensed globally, the risks are no longer theoretical. They translate daily into misdiagnoses, inappropriate treatments, and entire patient demographics being excluded from the benefits of a technology designed to serve them.
Algorithmic Bias: When AI Learns to Discriminate
The most dangerous aspect of medical AI is not merely that it makes mistakes, but that it errs systematically against specific communities. AI models learn from historical datasets that inherently reflect the biases of the healthcare system itself. Consequently, diagnostic tools that appear accurate on average can prove catastrophic for marginalised groups.
The numbers reveal the sheer magnitude of the problem. An analysis of 691 medical devices approved by the US Food and Drug Administration (FDA) demonstrated that 95.5% lacked demographic transparency, while only 1.6% possessed evidence from purely randomised trials. Worse still, a study published in PLOS Digital Health in 2026 found that out of 1,357 approved AI-backed devices, only 34 devices (2.5%) underwent registered clinical trials, and a mere 3 devices (0.2%) reported outcomes on actual patients. This means that the vast majority of devices currently deployed to make critical medical decisions have never been tested on the real-world patients they treat.
Furthermore, the few available trials are concentrated in single academic medical centers or resource-rich health systems, leaving performance in community hospitals, rural clinics, and low-resource environments entirely unverified. As representative bias becomes the dominant failure mode in AI diagnostic tools, the immediate consequences are missed diagnoses among underrepresented populations, inappropriate treatment recommendations, and delayed life-saving interventions.
Machine Hallucinations: Fabricating Medical "Facts"
If bias represents a structural flaw, "hallucination"—the fabrication of plausible-sounding yet entirely false information—is the most pressing hazard in the new generation of Large Language Models (LLMs). Documented cases have already emerged where patients used ChatGPT to self-diagnose, resulting in erroneous recommendations that caused permanent neurological injury and paralysis.
Even the most advanced models are not immune. Testing revealed that ChatGPT Health answers complex medical questions with the same unyielding "confidence" as simple queries, citing entirely fabricated medical studies and delivering misleading advice in nearly 50% of tested cases. According to the 2026 ranking by the independent patient safety organization ECRI, the "misuse of AI-powered chatbots in healthcare" was named the number one health technology hazard of the year.
The issue runs deeper than simple "errors"—it is a matter of profound public trust. Patients gravitate toward these tools because they project an aura of authoritative expertise. A 2026 global survey revealed that nearly 90% of individuals using AI trust the health insights provided on almost equal footing with their doctors, with 64% willing to share their entire medical history with AI for a faster diagnosis. This blind faith, in the absence of binding regulation, creates a perfect recipe for disaster.
Legal Liability: Who Holds the Machine Accountable?
When AI fails, the legal system faces a predicament. Who bears responsibility? Is it the physician who relied on the algorithm's recommendation? The hospital that purchased the system? The corporation that developed it? Or the owners of the data used to train it?
This legal ambiguity is no trivial detail; it stands as the primary obstacle to safe implementation. Across the WHO European Region, 86% of nations reported that "legal uncertainty" is the single greatest barrier to adopting AI in health. Academic studies consistently classify "unresolved legal liability" alongside systemic bias, the "black box" nature of complex models, and data privacy risks as core challenges.
This legal vacuum does not only harm patients; it undermines confidence in the entire healthcare ecosystem. A 2026 survey found that 75% of patients worry about accountability if AI contributes to harm during their medical care. When patients cannot identify who is accountable for a clinical error, they lose trust in everyone in the room—including the human doctor who committed no fault.
Regulatory Response: Steps in the Right Direction, Yet Far From Enough
It is not a total absence of regulation, as international and national bodies have begun to react, though technology advances at a far faster pace. In late 2025, the World Health Organization (WHO) issued strict ethical and governance guidelines for large language models in healthcare, warning against "technological solutionism"—the misguided belief that AI is a "magic bullet" capable of solving deep social, economic, and structural barriers. The guidelines outlined over 40 recommendations demanding mandatory ethical impact assessments for developers, robust privacy protections, and ensuring model design involves caregivers and patients themselves, emphasizing that AI systems "must prove accuracy and consistency in their responses before being placed between a doctor and a patient."
The US FDA has also initiated movement, though the regulatory safety net remains full of holes. In August 2026, the agency published a discussion paper on regulating generative AI medical devices, proposing a two-axis risk-evaluation framework: one axis assessing device autonomy and the other assessing the consequences of relying on erroneous outputs. However, the problem remains that this is merely a "discussion paper" seeking public comments, not a binding regulation. More dangerously, health chatbots like ChatGPT Health continue to operate in a regulatory grey zone: they offer diagnostic and therapeutic advice to users while evading regulation under the claim that they are "not medical devices"—even when behaving precisely as medical devices.
What Must Be Done? A Non-Negotiable Risk Framework
The greatest danger is not that AI will replace doctors. The true danger is that we will allow it to operate alongside doctors without safeguards, leaving patients stranded between two systems: an accountable human system, and an automated system subject to no accountability at all.
The response must occur across three distinct layers. First, mandatory, non-optional regulation: AI tools providing diagnostic or therapeutic advice must adhere to the exact same standards as medical devices, regardless of whether they are developed by a nimble startup or a tech giant. Second, radical transparency: patients cannot be asked to trust a "black box" system when they do not know how it functions, who trained it, or on what data. Third, patient involvement in design and governance, as recommended by the WHO, because those who bear the burden of errors must have a voice in preventing them.
In the end, this is not a technical question; it is a question of human values. Do we want artificial intelligence to make medicine more precise and equitable, or do we want technology that deepens inequality faster and renders it harder to detect? The answer is not yet decided. But it will only be resolved in favor of patients if we stop celebrating promises and start confronting risks—before those risks become part of daily medical routine.
Read Heba El-Baz's Articles

Thank you for your encouraging comment. Please continue to follow all the latest updates on the site. Regards, The Earn News Team.
Can't post here? Comment in a new window