In a new guidance document discussing the clinical testing of medical devices, the FDA includes a long section about the value of exploratory testing:
The Importance of Exploratory Studies in Pivotal Study
Medical devices often undergo design improvement during development, with evolution and refinement during lifecycles extending from early research through investigational use, initial marketing of the approved or cleared product, and on to later approved or cleared commercial device versions.
For new medical devices, as well as for significant changes to marketed devices, clinical development is marked by the following three stages: the exploratory (first-in-human, feasibility) stage, the pivotal stage (determines the safety and effectiveness of the device), and the postmarket stage (design improvement, better understanding of device safety and effectiveness and development of new intended uses). While these stages can be distinguished, it is important to point out that device development can be an ongoing, iterative process, requiring additional exploratory and pivotal studies as new information is gained and new intended uses are developed. Insights obtained late in development (e.g., from a pivotal study) can raise the need for additional studies, including clinical or non-clinical.
This section focuses on the importance of the exploratory work (in non-clinical and clinical studies) in developing a pivotal study design plan. Non-clinical testing (e.g., bench, cadaver, or animal) can often lead to an understanding of the mechanism of action and can provide basic safety information for those devices that may pose a risk to subjects. The exploratory stage of clinical device development (first-in-human and feasibility studies) is intended to allow for any iterative improvement of the design of the device, advance the understanding of how the device works and its safety, and to set the stage for the pivotal study.
Thorough and complete evaluation of the device during the exploratory stage results in a better understanding of the device and how it is expected to perform. This understanding can help to confirm that the intended use of the device will be aligned with sponsor expectations, and can help with the selection of an appropriate pivotal study design. A robust exploratory stage should also bring the device as close as possible to the form that will be used both in the pivotal trial and in the commercial market. This reduces the likelihood that the pivotal study will need to be altered due to unexpected results, which is an important consideration, since altering an ongoing pivotal study can increase cost, time, and patient resources, and might invalidate the study or lead to its abandonment.
For diagnostic devices, analytical validation of the device to establish performance characteristics such as analytical specificity, precision (repeatability/reproducibility), and limit of detection are often part of the exploratory stage. In addition, for such devices, the exploratory stage may be used to develop an algorithm, determine the threshold(s) for clinical decisions, or develop the version of the device to be used in the clinical study. For both in vivo and in vitro diagnostic devices, results from early clinical studies may prompt device modifications and thus necessitate additional small studies in humans or with specimens from humans.
Exploratory studies may continue even as the pivotal stage of clinical device development gets underway. For example, FDA may require continued animal testing of implanted devices at 6 months, 2 years and 3 years after implant. While the pivotal study might be allowed to begin after the six month data are available, additional data may also need to be collected. For example, additional animal testing might be required if pediatric use is intended. For in vitro diagnostic devices, it is not uncommon for stability testing of the device (e.g., for shelf life) to continue while (or even after) conducting the pivotal study.
While the pivotal stage is generally the definitive stage during which valid scientific evidence is gathered to support the primary safety and effectiveness evaluation of the medical device for its intended use, the exploratory stage should be used to finalize the device design, or the appropriate endpoints for the pivotal stage. This is to ensure that the investigational device is standardized as described in 21 CFR 860.7(f)(2), which states:
“To insure the reliability of the results of an investigation, a well-controlled investigation shall involve the use of a test device that is standardized in its composition or design and performance.”
This is what I’ve been arguing for a couple of years, now. If you want to test a medical device very well, then you have to test it in an exploratory way. This prepares the way for what the FDA here calls the “pivotal study”, which in software terms is basically a scripted demonstration of the product.
Yes, the FDA says, earlier in this guidance document, that it is intended to apply to clinical studies, not necessarily bench testing. But look at the reasoning: this exact reasoning does apply to software development. You might even say it is advocating an agile approach to product design.
Product development is iterative and incremental, has been for decades. Products are too complicated to develop them once, in a linear fashion. It’s nice that the FDA explicitly recognises that for medical devices.
[James’ Reply: Yes, that’s right. It is iterative and incremental BECAUSE we must learn as we go. Any methodology that outlaws or impedes learning cannot work. In other words, if you are using a methodology that impedes learning, then either A) you aren’t really using it, or B) you suck at your job.]
“Medical devices often undergo design improvement during development, with evolution and refinement during lifecycles extending from early research through investigational use, initial marketing of the approved or cleared product, and on to later approved or cleared commercial device versions.”
Anyone who’s involved in product development would expect this but too many people still want to criticise iterative, incremental approaches in software development by analogy with some fantasy of what product development is like. (Plus they often get product development confused with manufacturing).
So, that’s great. But is this FDA advice really recommending exploratory testing in the sense that software people mean when it talks about the phase called “exploratory”? That’s not clear to me.
[James’ Reply: I don’t know what every software guy thinks when he says “exploratory testing.” I just know what I mean by it, and I’ve been using the term longer than anyone except the man himself who coined it… And what the FDA is writing here is exactly my argument. They are warning against premature formalization of testing (as well as premature formalization of the product itself).]
I just don’t think this documents says what you want it to say.
Section 8.4 states:
“In a prospectively planned study a pre-specified protocol is used. Such a protocol would pre-specify study design, including inclusion/exclusion criteria, method of subject recruitment and selection, testing protocol, and analysis methods to be used.”
The testing protocol is to be pre-specified. Said protocol could well state that some exploratory testing should go on. But I’ll bet you a dollar that the vast majority of the testing that goes on in the vast majority of cases covered by this note is scripted. That occurs within an exploratory framework, which is good. The FDA are emphasizing to product developers that they should continue to learn about and change and improve their product all the way through its clinical trials, which is a good message.
[James’ Reply: You are taking that out of context. That is part of section 8, which covers Diagnostic Clinical Performance Studies. Look at the start of that section:
“For diagnostic devices, the pivotal clinical study is often a diagnostic clinical performance study. In such a study, clinical performance of the diagnostic device is characterized by clinical performance measures that quantify how well the diagnostic device output agrees with a subject’s true status, that is, how well it identifies, quantifies, detects or predicts an event or target condition as determined by a clinical reference standard.”
I’m involved in a project like that, today (see my blog entry about Paired Exploratory Surveys). As a matter of fact, I designed the formal accuracy testing protocol that they are using. But I designed it using ET (in other words, ET preceded the formal testing) and I continue to use ET in parallel with that formal process. In other words, exactly as the FDA states, exploratory study is a necessary precursor to non-exploratory study.]
But I don’t think this is arguing in favour of exploratory testing, unless exploratory testing is such a broad umbrella term that it’s pretty much useless.
It feels now as if I’ve intruded into some private argument: who are you trying to convince? Not me, I hope. I already think that exploratory testing is a good thing, in software development and elsewhere. I just don’t think this document is about that.
[James’ Reply: The document is exactly about that, whether the authors were thinking that way or not. Their logic is my logic, too, which makes it hard for them to repudiate my use of ET in medical projects.
Perhaps you are confused because you think I’m arguing in favor of using ET to the exclusion of any other approach. That’s not the case. What I’m happy about is that the FDA is admitting that we MUST use ET to get to the point where we can design an appropriate formalized study.
If this seems like a private argument to you, then perhaps you have not been in the position of having to justify ET to an executive who is puzzled that we would suggest doing ANYTHING not mandated by the FDA.]
James,
I believe your reasoning is correct.
You are constructing a thoughtful, logical, intentional, and defensible position for the use of an exploratory approach during testing. A project could use this logic and phrasing to describe and explain iterative development and the exploratory stage and elaborate on ISO 62304 using the FDA’s own words.
I note that section 4.2 has phrasing to the effect that the FDA will evaluate the device design sufficiently to understand device functions and mechanism of actions such that the agency is able to assess how and why the device works. To my ears, the FDA is restating the requirement that design is understood, and that understanding be supported by evidence.
I would assert that design understanding requires iteration and exploration of the device to generate good information to support the assertion. In my opinion, design understanding would improve if the designers adopted Karl Popper’s philosophy of, “a (design) theory gains creditability as it is subjected to (and passes) harsh tests”.
James,
Thanks you very much.
There is an updated link to the guidance link:
http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm373750.htm#s5