Q: We recently had a consultant come in to provide an overall look at our new shop floor execution system. The consultant declined to comment on the validation of this COTS (commercial off-the-shelf) application because we did not have table names, file structures, table mapping, relationship links, and data input linkage to tables. We do have a context diagram on how we use the application, and we’ve done extensive validation on the application with a full complement of validation documentation (URS, test scripts and results, validation plan and report, GxP risk assessment, etc.). The application is configured, but not customized.
I was of the understanding that, for purchased applications, this level of detail with table mapping and names, etc., is not required to demonstrate the application works as expected. We have audited the vendor and found them to use strong processes and good documentation.
So my question comes down to this: how important is it that we capture in our detailed specifications the following: table names, file structures, table mapping schema, relationships/links, data input mask linkage to tables, and output forms/templates?
A: The system you described would not be considered a COTS system by FDA due to its highly customizable nature (through myriad configuration choices). Instead, it would be considered a configurable system — one that, for optimum configurable system validation, you are expected to gather a great deal of data to support its use. Generally, that means significant auditing and conversing with the system provider, as well as evaluating the provider’s software development.
You indicate you have audited the vendor and qualified them as a software provider. That’s good — and the more data you can get on the vendor’s development and maintenance procedures and processes, the better for configurable system validation. As a practical matter, however, this is a product you are buying and configuring, so most likely you will never get all the data you might want.
The basic regulatory requirement for configurable system validation — like all systems — is that you must validate to your intended use. To do that, you need assurance the software was developed and will be maintained correctly, because ultimately you will be held responsible for its adequacy in meeting your needs.
[pullquote align=”right” cite=”” link=”” color=”” class=”” size=””]Especially for configurable systems, you must qualify vendors by evaluating (and finding adequate) their development and maintenance processes.[/pullquote]
Especially for a configurable system like this, you must qualify vendors by evaluating (and finding adequate) their development and maintenance processes. You also must define in detail your uses of the software and create tests to demonstrate the full successful execution of the software in your system. In setting up the system, each configuration choice must be treated as a “requirement” for the system (and documented as such) and must be tested objectively to demonstrate that the final configuration works as intended. All of this must be documented through the formal validation process you described in your question.
So, getting back to the heart of your question: if you can get the “table names, file structures, table mapping schema, relationship/links, data input mask linkage to tables, and output forms/templates”, it would be great to add all that information to the data you used to qualify the purchase of the software. But recognize that since the vendor — not you — maintains the software, the value of this data essentially will end with your purchase and the vendor’s first maintenance activity.
If, however, you create tables and populate them, etc., that information can be critical to how you maintain your overall system (hardware, software, people, processes and data) and therefore must be maintained and documented.
For a configurable system, the level of detail you describe — assuming you can get it — should be used for determining system qualification, but generally has little on-going value. If you can get the information, great, but most importantly continue to focus on your formal configurable system validation efforts and include your customization/configurations as you deploy the application.
Answered by Martin Browning, President and Co-Founder of EduQuest (22 years as an expert FDA investigator and as the Special Assistant to FDA’s Associate Commissioner of Regulatory Affairs, where he co-wrote the original Part 11 rules). Martin also is the developer and co-instructor of EduQuest’s FDA Auditing of Computerized Systems and Part 11/Annex 11 training class.