COTS Software System Validation: Can You Rely Just on Vendor Documentation?

Q: In 2016, I attended the EduQuest class on FDA Auditing of Computerized Systems. During the class we discussed the importance of clinical study sponsors to validate computerized systems used to satisfy (or assist with) specific business processes.

In June 2017, FDA issued draft guidance on the Use of Electronic Records and Electronic Signatures in Clinical Investigations Under 21 CFR Part 11 – Questions and Answers.

The new document includes the following sentence (lines 194 through 197):

“For COTS [Commercial Off-the-Shelf] systems that perform functions beyond office utilities, such as COTS EDC systems, validation should include a description of standard operating procedures and documentation from the vendor that includes, but is not limited to, results of their testing and validation to establish that the electronic system functions in the manner intended.”

Could you share EduQuest’s interpretation of this particular statement? Specifically, I’m interested in your thoughts (and the FDA’s guidance) on the extent to which a sponsor can rely on a vendor’s documentation, testing, and validation.

A: First, it’s important to note that this is Draft Guidance. It’s being distributed “for comment purposes only.”

This means FDA is setting forth a “strawman” guidance document for the purpose of soliciting public comment.

Nothing in the document is final until after FDA’s review of the comments and the final publication of the finished guidance. In addition, it’s intended to be a “guidance” document and therefore does not and will not contain requirements.

Now let’s address your specific question regarding FDA’s statement in lines 194 through 197:

This sentence sets forth some of what must be included in the “validation documentation” of a COTS system. It’s intended to reinforce proper due diligence in choosing COTS-based software for use in the clinical environment.

“Clearly, documentation from the vendor is not by itself totally sufficient.”

Note that it says “validation should include…” — not “only consist of”.  Clearly, the documentation from the vendor is not by itself totally sufficient.

Additional validation documentation is needed to demonstrate the COTS software is adequate to fulfill the user’s “conditions of use”; demonstrate fit and function within the user’s defined procedures and processes; and show that personnel have been trained in its use, etc.

Validation should always include at least these two elements:

1)  Show that good software development practices were used to create the software, and

2) Show that the software functions properly in the user’s targeted environment.

The first element is problematic for the user of COTS software because of the lack of visibility into the developer’s software engineering processes.

FDA recognizes this limitation. As result, I believe this draft guidance is FDA’s attempt to define the minimal documentation a user needs from the developer to demonstrate that the user has applied due diligence in choosing COTS software.

The second element remains the sole responsibility of the user of the COTS software. Only your own efforts and documentation can meet that expectation.

Unfortunately, FDA’s draft guidance does not address what should happen when the COTS software developer will not share the requested information with the user. Perhaps the Agency will address that situation in future guidance.

Answered by Martin Browning, President and Co-Founder of EduQuest, who co-wrote the original 21 CFR Part 11 regulation during his 22-year career at FDA. Martin is the chairman and lead developer of EduQuest’s FDA Auditing of Computerized Systems and Part 11/Annex 11 Compliance training class. This class was originally developed to train FDA’s own investigators how to audit computerized systems, then was modified for industry. During the past 21 years, it has trained thousands of auditors, quality managers, and IT specialists in FDA-regulated companies.


Leave a Reply

Your email address will not be published. Required fields are marked *