Imagine overhearing your doctor in the hallway outside the exam room, phone in hand, quietly asking a free AI chatbot which medication to prescribe for your chronic condition.
Now imagine a different scene. Your physician explains that your hospital’s AI system has reviewed your recent test results, insurance coverage, medications over the last 10 years, any food, financial or transportation insecurities that impact your care journey, and the latest clinical guidelines. Together, you discuss the options and decide on a course of treatment that fits your needs.
Both futures are possible. You are likely to face the first, but too few hospitals can offer you the second option right now.
Across the United States, physicians are increasingly turning to “free”” AI to manage the growing demands of modern medicine. Administrative burdens have ballooned. Burnout is widespread. Time with patients is scarce. And their health system does not have the resources to integrate an institutional AI in the way that you would expect and want for your family. Too often, doctors respond by using generic, free AI tools on their own, without institutional oversight or support.
That’s the making of a national crisis that could split the public between those who can afford the most expensive, AI-driven health systems and the rest, who would be left to fend for themselves.
When the software’s free, you’re the product
This practice, commonly referred to as “shadow AI,” is no longer rare. A recent survey found that a majority of frontline health care workers now use generic, free AI solutions for work at least once a month, and almost 40 percent used AI at least once a week. Another survey found that 10 percent of healthcare professionals acknowledge using AI in direct patient care, shaping diagnoses, treatments, and follow-ups.
The risk is not that AI exists in medicine. The risk is how it is being used.
When artificial intelligence quietly enters the exam room without clear governance, responsibility becomes blurred. Who validates these tools? Who monitors them for errors or bias? Who is accountable if something goes wrong? In health care, those questions are not academic. They are central to patient safety and public trust.
The problem with shadow AI is not that large language models make mistakes. All complex systems do. The deeper concern is that these mistakes happen outside the safety structures that define modern medicine.
Shadow healthcare
A free, consumer-facing AI tool is not validated using local patient populations. It is not monitored over time for performance changes. It is not audited for bias or data security. It does not feed into institutional quality systems that allow organizations to learn from errors and prevent harm. In effect, the AI behaves like a medical device without being treated as one.
Recent guidance from the Joint Commission and the Coalition for Health AI underscores this point. Responsible clinical AI requires formal governance, multidisciplinary oversight, validation within local workflows, and ongoing monitoring for safety and bias. Artificial intelligence does not replace human judgment. It increases the need for it.
Shadow AI provides none of those protections.
Some argue that the solution is to ban informal AI use. Many physicians rightly note that prohibition would only push the practice further underground. Both views miss the larger issue. The solution is not banning AI or pretending it is not already here. It is enabling hospitals, rather than individual clinicians, to deploy and govern AI openly and responsibly.
There is a reason shadow AI has flourished. Hospitals face shrinking reimbursement, staffing shortages, and relentless administrative complexity. Properly integrating AI across clinical workflows requires significant upfront investment, technical expertise, and ongoing oversight. Many health systems, especially smaller and rural hospitals, simply cannot afford that on their own.
Free software tools are rarely designed with equity in mind and may reflect narrow training data or individual user behavior. Over time, this risks creating a parallel system of care that is fragmented, inconsistent, and increasingly shaped by institutional wealth rather than medical need.
Large academic medical centers may be able to meet emerging standards for responsible AI use. Community hospitals that serve many of the nation’s most vulnerable patients often cannot. Without deliberate intervention, artificial intelligence threatens to widen existing disparities rather than reduce them.
Health care already relies on shared systems for physician licensing, accreditation, and safety oversight. Artificial intelligence deserves similar collective infrastructure. Expecting thousands of hospitals to independently validate and monitor complex algorithms is a recipe for duplication, gaps in safety, and uneven care.
Today, hospitals are being asked to manage AI in a regulatory vacuum. Federal oversight remains fragmented, and most clinician-assisted tools fall outside existing frameworks. That gap is one reason shadow AI continues to grow.
False promises
Critics who think I’m exaggerating the threat should consider how many technologies that promised to improve products ultimately could have a dark side if not implemented effectively.
Online education has its rewards, but it can be a bad substitute for in-class experience.
The internet was supposed to democratize information, yet it’s been blamed for devastating the country’s journalism industry.
The gig economy offers flexibility but also delivers precarity.
If trust, transparency, and accountability remain core values in medicine, AI governance cannot be left to improvisation.
We need a bigger conversation about how effective government leadership, healthcare industry leaders, public policy leaders, and AI entrepreneurs can help all our institutions become the driving force behind healthcare innovation that’s chosen, not resorted to in the shadows.
When AI operates quietly and without oversight, it erodes trust, obscures responsibility, and risks creating a healthcare system that works well only for those who can afford it.
The future of medicine is already here. The question is whether we choose to build it deliberately and in the open, or allow it to take shape in the shadows. Public trust in medicine has always rested on judgment, candor, and accountability. Artificial intelligence can reinforce those values, but only if hospitals are empowered to lead.
Photo: ismagilov, Getty Images
Deepthi Bathina is the Founder and CEO of GW RhythmX, an AI-native company defining the category of Enterprise Precision Care AI and building the large-scale foundation for the next generation of intelligent, connected Smart Hospitals. Formed through the merger of Get Well and RhythmX AI, the platform is deployed across more than 150 health systems, reaching over 85 million patients including 8 million U.S. military veterans and is powered by one of the industry’s deepest healthcare datasets spanning 300 million patient records and 4.4 billion annual claims. Prior to founding RhythmX AI, Bathina held senior executive roles across the health technology industry, including Chief Product Officer at Humana, and COO and global leadership roles at Nuance Communications and Wolters Kluwer Health. She has led global product, operations, and technology organizations at scale, managing multi-billion-dollar P&Ls and operating across 170+ countries.
This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.
