And with the apparent inclusion of material imagined by AI, referred to as hallucinations, in the “Make America Healthy Again” report, they see an ominous sign for the administration’s ability to deploy it safely.
“If they were proceeding more apace, and I’m not talking about glacial government pace, I’m talking about responsible pace, this wouldn’t happen,” said Oren Etzioni, a professor emeritus at the University of Washington and entrepreneur who studies AI. “The presence of these embarrassing missteps just shows that it’s amateur hour.”
Kennedy has expressed ambitious but vague plans, usually in the context of cutting costs. He has outlined his vision during several congressional hearings, including one to replace the use of animals in experimental testing and some steps in clinical trials.
“We’re phasing out most animal studies . . . because we can accomplish a lot of those goals on safety and efficacy with AI technology,” Kennedy said.
He also mentioned using AI to analyze data that HHS and other agencies have collected on patients, such as people on Medicare and Medicaid. He said they’ve recruited experts to “transform our agency for a central hub for AI.”
So far, the Food and Drug Administration has announced a tool named “Elsa” to summarize documents for FDA employees and released a plan to minimize animal studies for certain treatments and drug development by using testing from other countries, and computer and AI modeling.
“In the long-term (3-5 years), FDA will aim to make animal studies the exception rather than the norm for pre-clinical safety/toxicity testing,” the FDA said in its road map.
But the mistakes in the “Make America Healthy Again” report have experts skeptical of HHS’s ability to use AI correctly. The report cited sources that do not exist and had garbled footnotes, first reported by the online news publication NOTUS, and bore other hallmarks of AI, The Washington Post reported. Those issues were later corrected
The agency declined to answer definitively whether AI was used on the report and why it contained nonexistent citations, but rather only highlighted the substance of the report. It also did not offer specifics about protocols for responsible AI use.
“HHS is addressing the risk of AI-generated errors through rigorous validation, human oversight, and strict quality controls,” a spokesperson said in a statement. “AI tools are designed to support — not replace — expert judgment.”
Experts say the specifics of how AI is implemented will bethe true measure of whether the efforts at HHS will succeed or end up being harmful.
“I’m actually deeply optimistic about what [AI] can do in a lot of areas, including the ones that the secretary mentioned,” said Ziad Obermeyer, a physician and researcher at the University of California Berkeley who studies AI in biomedicine. “What my research has shown is that it actually comes down to some of the really boring details that make the difference between a good, powerful algorithm that helps people, and one that really messes things up.”
Republicans who work on AI issues in the Senate supported Kennedy’s goals but also agreed on the importance of rolling it out with the right protections.
“This is going to be the future,” said Indiana Senator Todd Young. “I mean, we’d be doing something wrong if, if the head of our health agency wasn’t talking about using AI.”
The two general use cases that Kennedy has mentioned, replacing steps in clinical trials and analyzing patient data, have some potential issues in common, including that AI can generate false information.
But they also have risks unique to each case. Privacy, for example, is a serious concern with patient data. If not properly stripped of identifying factors, even supposedly anonymized data can be re-identified, as has happened in some cases.
“The most secret private information that people have is their health care data, and so AI should not be used in any way that does not have the strongest possible safeguards,” said Massachusetts Senator Ed Markey, a Democrat. “We could have an absolute privacy catastrophe.”
Harvard Law School professor I. Glenn Cohen, who studies medical ethics and AI, said that the idea holds great potential, but that the administration would need to be very transparent about how it is protecting data and would be wise to run smaller pilot studies first.
“The ‘move fast and break things’ ethos of Silicon Valley may be appropriate in some parts of life — I don’t really care if you’re doing it for the order of Instagram postings,” Cohen said. “But it’s not a philosophy we advocate for physicians or an attitude I think most people want health care to take.”
The key limitation of AI is that it is only as good as the dataset used to build it. In specific areas where scientific data is really good and outcomes are predictable, such as in the structure of proteins, scientists have built powerful AI tools. AI can also help doctors and patients assess symptoms.
But those are different from the discovery of new information, experts say, which is what a lot of science and clinical trials for novel treatments are designed to explore.
Allison Coffin is a researcher at Creighton University who studies hearing loss, including that caused by certain medicines. She uses mostly zebra fish in her work, but also rodents. She says her lab is working on AI tools to help identify potential toxins in order to conduct more targeted research. But, she said, AI would always be used as an idea generator for testing in animals, not to replace them.
“That’s an excellent case for AI, because AI can rapidly assess millions of potential drug structures. But you would still want to test their efficacy for new therapies in an animal,” Coffin said. “I would never want to take a medication that hadn’t been given to a living creature before, and I would think most people wouldn’t. Do we want to be the first to take medication because a computer model says that it’s safe?”
Other scientists questioned the ability of the government to do the cutting-edge research necessary after the administration’s deep cuts to research funding and staff.
“Honestly I’m struggling for what to say,” wrote Sean Eddy, a Harvard scientist who works on building computer models for biology and genomic research. “I just don’t see how it makes sense for HHS to talk about delivering innovative technological breakthroughs while they’re destabilizing and belittling the US scientific research enterprise. . . . Every lab at Harvard that does this kind of research, including my own, has had all their federal funding terminated.”
Experts also question whether Kennedy understands where AI technology actually is today versus its potential capacities. Many cited cautionary tales of much lower-stakes AI deployment gone wrong, such as companies that fired staff and replaced them with AI now scrambling to rehire real people or a New York City chatbot that was advising companies to break the law.
“Doing things like simulating an entire body in order to save clinical trials is just grossly unrealistic where we sit right now,” said Gary Marcus, a professor emeritus at New York University and critic of AI enthusiasm. “If we’re lucky, we can do it in 40 [years], but we certainly can’t now. That’s just a pipe dream.”
Tal Kopan can be reached at tal.kopan@globe.com. Follow her @talkopan.
link