Breaking Down the FDA Guidance: Human Factors in AI-Enabled Medical Devices Software

Introduction to human factors in AI-enabled medical device software

Artificial Intelligence (AI) in medical devices has introduced new possibilities, but it also brings new usability challenges. The FDA’s draft guidance on AI-enabled medical devices highlights the importance of human factors (HF) engineering throughout the total product lifecycle.

If you want to learn more about human factors in the medical device or pharmaceutical industry, I recommend checking our article, “Introduction to Human Factors in Medical Devices and Combination Products.”

In the following sections, I want to share some of my own observations on the new FDA AI guidance from the perspective of a human factors professional. The guidance has generated a lot of discussion in the industry, and I think the attention it gives to human factors is particularly interesting.

The FDA’s perspective on human factors in AI-enabled devices

As a human factors professional, one thing that immediately caught my attention is that the guidance references the term “Human Factors” 34 times, including some footnotes, and “Usability” another 34 times, although not always in the same context.

It is not common to find an FDA guidance outside the traditional HF documents making such frequent and deliberate references to human factors and usability.

For me, this says two things. First, it recognizes the role of human factors in ensuring the safety and effectiveness of medical devices. But it also shows how seriously the FDA is considering usability in the context of AI-enabled devices. I did not entirely expect this, especially considering how permissive the FDA has sometimes been around human factors for software-based devices, as we have seen in the premarket device-software guidance. So I am happy to see this insistence on usability and safety.

Going back to the guidance itself, the FDA emphasizes the need to consider how different users, from clinicians to patients, will interact safely and effectively with AI-enabled devices. There are a few areas that I think are particularly relevant.

1. User Interface (UI) Design: Regulatory submissions should include detailed descriptions of the user interface, including graphical elements, operational sequences and user workflows. Appendix B of the guidance gives particular attention to this.

I think this is important because users need to understand what an AI-enabled device is telling them and what they are expected to do with that information. The interface is part of that communication. If the AI output is difficult to understand or interpret, that can directly affect decision-making and, ultimately, safe and effective use.

2. Usability Validation: Usability evaluations should demonstrate that intended users can interact with the device safely and effectively. This means considering differences between users, use environments and realistic use scenarios rather than assuming that everyone will interact with the technology in the same way.

The guidance also emphasizes human factors validation testing for critical tasks, meaning tasks where use errors could result in serious harm. This is consistent with the human factors approach already used for medical devices: identify the use-related risks, design them out or reduce them where possible, and validate that the final design supports safe and effective use.

3. Risk Mitigation: Risk assessments need to consider how people interact with the device, including the possibility that users may misunderstand or incorrectly interpret information generated or presented by the AI.

The FDA guidance prioritizes design solutions over relying primarily on training or labeling to control these risks. This is consistent with the general approach in ISO 14971 and IEC 62366. The preference should be to address the problem through the design whenever possible rather than expecting instructions or training to compensate for a difficult interface.

Validation activities for AI-enabled medical devices

According to the FDA, validation of AI-enabled medical devices needs to demonstrate that the device can be used safely and effectively. From a human factors perspective, this is not really new. It is the same basic expectation presented in the FDA human factors guidance and reflects the process the medical device industry has been following for more than a decade.

What I find more interesting is the distinction the guidance makes between human factors validation and usability evaluation.

The guidance states that “usability describes whether the device can be used safely and effectively by the intended users, including whether users consistently and correctly receive, understand, interpret, and apply information related to the AI-enabled device.”

At the same time, it treats usability evaluation and formal HF validation somewhat differently.

In my view, this suggests that the FDA expects manufacturers to have a usability engineering process even for devices that do not have critical tasks requiring a formal human factors validation study.

The guidance also references the FDA’s main human factors guidance, “Applying Human Factors and Usability Engineering to Medical Devices,” which is basically the bread and butter for any human factors practitioner.

That guidance is mainly focused on devices with critical tasks, but the FDA points out that its recommendations can also be useful for demonstrating appropriate control of other use-related risks. I think this is an important point. Not having critical tasks does not mean that usability engineering disappears from the development process.

Usability to support control of risks in AI-enabled medical device software

In Appendix D, the FDA makes this quite clear. Even when a device does not have critical tasks associated with its use and the manufacturer does not need to submit an HF validation report, the manufacturer should still consider following the practices described in the FDA human factors guidance to evaluate the usability of the AI-enabled device.

This is where things become particularly interesting. In the absence of a required HF validation, the HF process can still be used to demonstrate the effectiveness of proposed risk control measures. We already do this quite frequently in the industry.

IEC 62366 also requires the usability engineering process to address risk control measures related to the user interface. When labeling is part of those controls, the relevant information needs to be identifiable, understandable and capable of supporting safe and effective use.

I must admit that I am a little concerned about this area. I think the approach makes sense, but I can also imagine some very lengthy human factors evaluations, particularly while the technology is still immature and manufacturers continue to rely heavily on labeling-based mitigations.

This is where I think manufacturers need to involve human factors practitioners early. The important question is not only how a mitigation will eventually be validated, but whether it has been designed in a way that can realistically be validated in the first place.

I would recommend that manufacturers discuss these issues early with experienced HF professionals and think carefully about how risk mitigations are defined.

Reflections on the role of human factors in the new FDA guidance for AI-enabled medical device software

For me, one of the clearest messages from the new FDA guidance is that human factors and usability engineering are not limited to AI-enabled devices with critical tasks.

Formal HF validation may not always be required, but that does not remove the need for a usability engineering process. Manufacturers still need to understand how intended users interact with the device, where use-related risks may appear and whether the proposed controls actually work.

This is also why I think human factors practitioners should be involved early in the development of AI-enabled medical devices. Waiting until validation to discover that a risk control depends on information users do not notice, understand or apply correctly is a difficult problem to fix.

The way those controls are designed will have a direct impact on what eventually needs to be evaluated or validated. With AI-enabled devices, where interpretation of information is often central to the user interaction, I expect that relationship between risk management and human factors to become even more important.