Autonomous Tech in Healthcare Raises Urgent Legal Questions, Attorneys Warn

As "agentic AI" moves from science fiction to clinical reality, legal experts caution that the technology is outpacing the law, creating a minefield of liability and patient safety concerns.

A new frontier of artificial intelligence, known as "agentic AI," is beginning to move into the healthcare sector, promising to automate complex tasks, from managing prescriptions to potentially even aiding in diagnostics with minimal human oversight. This evolution from AI as a tool to AI as an autonomous actor is creating a significant legal gray area, according to legal professionals who warn that our current frameworks are unprepared for the complex challenges ahead.

Agentic AI systems are not just responsive; they are proactive. They can be designed to take independent action to achieve a set of goals, a capability that distinguishes them from more familiar AI like ChatGPT. In healthcare, this could mean an AI system that not only flags a potential drug interaction but also cancels the prescription, alerts the physician, and schedules a follow-up appointment, all without direct human intervention.

While the potential to increase efficiency and reduce physician burnout is immense, the legal implications are just as profound. According to Lily Li, founder of the law firm Metaverse Law, this shift to autonomous action removes the human from critical decision-making loops, which can have life-or-death consequences. "If there are hallucinations or errors in the output, or bias in training data, this error will have a real-world impact," she stated in a recent interview.

Legal experts are highlighting several key areas of concern:

Medical Malpractice and Liability: When an autonomous AI is involved in a patient's care, who is responsible if something goes wrong? Is it the hospital that deploys the AI, the developer who created the software, or the physician who uses it? Attorney Meghan O'Connor of Quarles & Brady points out that it will become increasingly difficult to apportion negligence between the AI software, the medical device it's integrated with, and the human healthcare provider. The fundamental question becomes: what is the standard of care for an AI? Should it be held to the same standard as a "reasonably prudent person" or a higher, more stringent one?

Patient Safety and Bias: AI models are trained on vast datasets, and if these datasets contain inherent biases, the AI can perpetuate or even amplify existing healthcare disparities. An AI trained predominantly on data from one demographic might be less accurate for others, leading to unequal care. This "black box" nature of some AI decision-making can make it difficult to identify and correct for such biases, posing a significant risk to patient safety.

Regulatory and Compliance Hurdles: Current healthcare regulations, such as HIPAA in the United States, were not designed with autonomous AI agents in mind. The use of AI to handle protected health information (PHI) introduces new privacy and security risks. Furthermore, it's unclear how regulatory bodies like the FDA will classify and regulate these advanced AI systems, which can learn and evolve over time.

Informed Consent: How can a patient give informed consent when their care is being managed or influenced by an autonomous AI? The traditional doctor-patient relationship is built on trust and communication. As AI takes on more agentic roles, new standards for transparency will be needed to ensure that patients understand how their data is being used and how decisions about their health are being made.

Attorneys and tech experts agree that for agentic AI to be adopted safely and ethically in healthcare, a multi-faceted approach is necessary. This includes developing robust governance frameworks, ensuring transparency in AI decision-making, and updating legal and regulatory standards to address these new challenges. As Li notes, the future of this technology in healthcare may depend less on its technical capabilities and more on the ability of the industry to "build trust and accountability." Without clear legal guardrails, the promise of a more efficient healthcare system could be overshadowed by the peril of navigating an undefined and potentially dangerous legal landscape.