FDA's Clinical AI Rollback Leaves Hospitals Without Transparency Standards

By the end of 2025, the FDA had cleared more than 1,450 artificial intelligence-enabled medical devices. In that same year alone, 295 new AI tools received authorization. Two-thirds of American clinicians now incorporate AI into their daily work. Yet the federal government is preparing to weaken the only transparency rules that govern how these tools are built, tested, and maintained inside the hospital systems that serve 96 percent of U.S. hospitals.

The numbers tell a story of rapid deployment without adequate safeguards. Fewer than 2 percent of those 1,450 cleared devices were supported by randomized clinical trials—the gold standard for medical evidence. Studies of large language models used to support clinical decisions show hallucination rates between 8 and 20 percent, meaning the systems generate plausible-sounding but false information at measurable rates. Against this backdrop, the Department of Health and Human Services released a proposed rule in January 2026 called HTI-5, which would eliminate the transparency requirements that were meant to keep pace with deployment.

Those requirements, finalized in late 2023, were modest but meaningful. They obligated developers of AI tools in certified electronic health records to disclose 31 standardized source attributes—categories describing how each tool was built, what data trained it, how it was tested for fairness, what its performance metrics were, and how it would be maintained over time. Developers also had to conduct intervention risk management, a structured process for identifying and mitigating potential harms across the tool's entire lifecycle. Together, these provisions represented the closest the United States had come to a model-card mandate, the kind of standardized "nutrition label" that would tell hospital administrators what they were buying.

HTI-5 would remove all of it. The agency justifies the rollback by arguing that the transparency requirements lack demonstrated clinical utility and that they conflict with the White House's broader push to deregulate AI. Some industry observers see relief in the change—fewer barriers to development and deployment. But the rollback rests on a fundamental misreading of what those transparency rules were designed to do. Regulators treated them as paperwork burdens imposed on clinicians at the bedside. In reality, the source-attribute regime was built for institutional buyers: hospital procurement officers, compliance leads, and quality committees who needed a standardized floor to compare vendors, audit deployed models, and meet their own governance obligations. Removing that floor does not eliminate burden; it shifts it onto the hospital, which must now construct its own diligence process for every algorithm in every workflow.

This shift arrives at a moment when two other regulatory gaps are widening. First, the FDA's own oversight, though broad in scope, was never designed to monitor how AI tools perform after they enter the market. A 2025 analysis of cleared devices found that most of the public evidence manufacturers file to obtain FDA clearance lacks basic details on study design, sample sizes, and whether the research included diverse populations. The FDA has acknowledged this gap and begun asking questions about real-world performance measurement and model drift—the phenomenon where an AI tool's accuracy degrades as patient populations change. But that work remains preliminary. In the interim, the source-attribute disclosure regime forced developers to publish validation and update schedules, filling part of the post-market accountability void. Removing it leaves that void exposed.

Second, much of the clinical AI now entering hospital workflows is not regulated by the FDA at all. Internally developed models, non-device clinical decision support tools, and generative AI assistants frequently fall outside the agency's jurisdiction. The transparency criterion was deliberately drafted to cover both FDA-regulated software and non-device predictive tools supplied through certified health IT systems. Eliminating that umbrella means that the fastest-growing category of clinical AI—generative tools built on foundation models—operates in a regulatory blind spot at the exact moment when hallucination rates remain measurable in the double digits.

A more defensible path would preserve the source-attribute disclosures and reframe them as one layer of continuous monitoring. The nine disclosure categories that were codified—purpose, development inputs, fairness process, external validation, performance measures, ongoing maintenance, update schedule, cautioned out-of-scope use, and intervention details—align cleanly with the AI risk management framework published by the National Institute of Standards and Technology and with the post-deployment surveillance questions the FDA itself is now asking. Rather than retiring the model card, regulators could require that those attributes be machine-readable, versioned, and exposed through a standard application programming interface. Hospital governance committees, payers, and accreditors could then verify them continuously. That approach treats AI governance as infrastructure rather than as a one-time disclosure event. It places the marginal cost of compliance on the developer who controls the model, and it gives the rest of the health care system a common substrate for oversight. It also acknowledges what three years of deployment have demonstrated: clinical AI may not fail at the moment of regulatory clearance, but it may fail quietly, over months, as populations shift and models drift.