Chapter 8. Perform Qualification Activities

Introduction

This topic looks at performing qualification activities, the fifth step in the validation process.

Types of reviews

These are the types of qualifications that can be performed:

·         Supplier audit (this would be applicable to both internal and external suppliers)

·         Source code review

·         Specification qualification

·         Design qualification

·         Installation qualification

·         Operational qualification

·         Performance qualification

Note

At the end of the development process for a bespoke application, the supplier will usually perform a System Acceptance Test at the supplier site.

A request should be made to obtain this documentation for inclusion as part of the validation documentation.

Introduction

The supplier audit is usually undertaken for configurable software packages or custom built/bespoke software. It should be performed either:

·         prior to the formal commitment to purchase (for configurable software packages)

·         during the development process (for custom built/bespoke software)

Pre-qualification of suppliers

For projects where either a number of suppliers could potentially offer a packaged solution or a supplier is being selected for a custom activity, then a number of suppliers may be subject to a ‘pre-qualification’.

The pre-qualification may take the form of a visit or a questionnaire, which is sent to the supplier for their completion. The questionnaire would contain questions relating to the supplier’s organization and quality management system.

Responsibility

The System Validation Team will either:

·         undertake the supplier audit, or

·         check the results from the supplier audit

Note: The System Validation Team will check the results from the supplier audit when they perform the Specification Qualification (SQ) (See, Specification Qualification later on in this chapter).

Purpose

The purpose of the audit is to assess the supplier or development group’s quality management system, specifically the development methodology and quality plan, to ensure that quality assurance is built into the software development process.

The audit should verify that the development methodology conforms to the system development standards specified as part of the validation process.

Recommendation

It is recommended that you use the following Good Automated Manufacturing Practice (GAMP) Guide as the standard against which suppliers are assessed:

GAMP - "Supplier Guide for Validation of Automated Systems in Pharmaceutical Manufacture" Produced by the GAMP Forum.

GAMP

Although the GAMP document has been written specifically with manufacturing systems in mind it is recommended as a general reference as the principles contained within it are general and can be applied to the validation of systems of all types.

Other references

You should also refer to:

·         ISO 9001: 1994 Quality Systems. Model for quality assurance in design/development, production, installation and servicing

·         ISO 9000-3: 1991 Guidelines for the application of ISO 9001 to the development, supply and maintenance of software

·         ISO 10011 Quality Systems Auditing
Part 1: Auditing
Part 2: Qualification criteria for Quality Systems auditors
Part 3: Managing an audit programme

·         ANSI/IEEE Standards (detailed reference may be found in the Reference Material Section)

·         The TickIT Guide - A guide to Software Quality System Construction and Certification using ISO 9001.

Audit details

The supplier audit should address the following topics:

·         the supplier’s development methodology and quality plan

·         the documentation trail of a key product or component through the development life cycle to verify that the methodology has been followed

·         project status reports or other internally generated documentation

·         evidence of good testing procedures

Testing

Testing should be performed concurrently during the development of the system by the developer of the software, not solely by the purchaser or user at the end of the development process.

If there is sufficient evidence of testing, the testing procedures review can be considered part of the Operational Qualification.

Introduction

This topic covers source code review.

When to review source code

Use the table below to determine if Company should perform a source code review:

If ...

Then a source code review will...

a supplier audit is not possible

be required

there is satisfactory evidence in the supplier audit that the source code was developed in a high-quality manner and subject to review as part of the development life cycle

not be required

a supplement to the supplier audit is desired

be required

Note: The source code review should be a high-level review. It should use diagrams and charts of the software.

Note

Source code should be characterized to identify:

·         key information, including version number and the language used

·         other detailed information, if it will add value to the validation exercise

Recommendation

It is recommended that you use the following Good Automated Manufacturing Practice (GAMP) Guide as the standard against which source code standards are assessed:

GAMP - "Supplier Guide for Validation of Automated Systems in Pharmaceutical Manufacture" Produced by the GAMP Forum.

Introduction

The Specification Qualification (SQ) is a technical, quality and commercial review of the tender/requirements package.

Note: Bespoke systems will require a full SQ whereas off-the-shelf systems will require a much simpler SQ.

Purpose

The purpose of the SQ is to show that the controls required to specify the design have been addressed and agreed by the user, and where appropriate the in-house implementation group.

Documents to be reviewed

The following documents may be reviewed during the SQ:

·         User Requirements Specification

·         Functional Specification (if available at this time)

·         Supplier Audit Report

·         Supplier Contract

·         Commercial and Purchasing Specifications

Acceptance criteria for the SQ will be defined in the Validation Protocol. The protocol will define which documents should be prepared, the standards they must meet and the approval status required.

Recommendation

It is recommended that you use the following Good Automated Manufacturing Practice (GAMP) Guide as the standard against which the above documents are reviewed:

GAMP - "Supplier Guide for Validation of Automated Systems in Pharmaceutical Manufacture" Produced by the GAMP Forum.

Supplier audit

If a supplier audit has not been done, then the validation team may be required to undertake one.

SQ Report

The results of the SQ should be documented in the SQ Report.

The report should also include:

·         details of tasks done during the SQ

·         supporting documentation

·         version numbers or dates of documents reviewed

Introduction

The Design Qualification (DQ) is a review of system design and documentation for technical content and quality. It is usually performed prior to installation at the site.

Note: Bespoke systems will require a full DQ whereas off-the-shelf systems will require a much simpler DQ.

Purpose

The purpose of the DQ is to:

·         confirm that the system development life cycle has been followed

·         ensure that the individual elements of the system have been designed and proven

·         ensure that the user and supplier agree that the integration of all the elements meet the User Requirements Specification, Functional Specification, Design Specification and Quality plan

Documents to be reviewed

The following documents may be reviewed during the Design Qualification:

·         Functional Design Specification

·         Flow Diagrams

·         System Diagrams

·         User Manuals

·         System Manager Manuals (Administration Documentation)

·         Design phase implementation and test documentation

·         Drawings

·         Material and equipment lists

·         Quality plans

·         A listing of any deviations from the User Requirements Specification for the system developed

·         Compliance matrix (i.e. list of functions with reference to their quality critical nature)

·         Change control records

·         Source code development review

Note: Although not all the documentation may be available, sufficient documentation must exist in order to verify the system at the DQ phase.

Some or all of the requirements of the DQ will be met if a supplier audit is performed.

Acceptance criteria for the DQ will be defined in the Validation Protocol. The protocol will define which documents should be prepared, the standards they must meet and the approval status required.

Recommendation

It is recommended that you use the following Good Automated Manufacturing Practice (GAMP) Guide as the standard against which the above documents are reviewed:

GAMP - "Supplier Guide for Validation of Automated Systems in Pharmaceutical Manufacture" Produced by the GAMP Forum.

DQ Report

The results of the DQ should be documented in the DQ Report. The report should also include:

·         details of tasks done during the DQ

·         supporting documentation

·         version numbers or dates of documents reviewed

Introduction

The Installation Qualification (IQ), is the process of demonstrating that the system hardware and software has been installed according to the manufacturer’s specifications and that all deliverables are properly accounted for.

The IQ is achieved by writing an IQ protocol and documenting the installation to ensure it meets the acceptance criteria defined in the protocol.

Note: For the purpose of this guide, the IQ covers both hardware, system software and application software.

System software

System software (the operating system) is validated by default (because the application will not function without the system software) when the application software is qualified and the system is validated.

It does not need to be validated independently of the application software.

However, you do need to qualify the installation of the system software, and the operation and maintenance of the software.

Qualifying new applications

When a new application is installed on a platform that has been validated for a different application, then an IQ only needs to be performed on the new application.

The new IQ should refer to the original IQ for the hardware and system software.

IQ Protocol

An IQ protocol should be written.  

The IQ protocol describes how the hardware and system and application software should be installed and should contain:

·         system description, functional and design specifications as appropriate

·         manufacturer’s documented recommendations for installation

·         manufacturer’s instructions for unloading and installation

·         post-installation procedure, if any

·         associated equipment (Note: associated equipment should be qualified)

·         associated conditions, e.g. environmental and ergonomic conditions

·         associated documentation and deliverables including but not limited to:

а         List of approved deliverables

а         Manuals, operations, administration, technical and maintenance

а         Patch notes and/or release notes

а         Structures list (database, fields, types, sizes, keys, links and referential relationships)

а         SOP List and SOPs (Application, Systems, Environment, Training, IT specific and Corporate Software/Hardware critical SOPs.)

а         Screen shots

а         File list (application, shared, 3^rd party installation,

а         Module or component list

а         Media list and storage locations

а         Report list

а         Registry changes list

а         Functional Requirements

а         User Requirements

а         Project Plans

а         Flow diagrams and/or schemas

а         User and Functional Matrixes

а         Pre-requisition descriptions and Matrixes

а         Training procedures and employee training records

а         Errors, dialogs and operator interface messages

а         Audits and audit findings

·         acceptance criteria

·         responsibilities and roles

Qualification scripts may be written to support the requirements of the protocol.

Acceptance criteria for the IQ will be documented in the IQ Protocol and the qualification scripts, and will be based on the functional and design specifications along with the installation instructions.

IQ results

The IQ results should include the following:

·         documentation of the installation procedure followed by the installer

·         notes made by the installation technician during installation

·         records of data entered or received from the system during installation

·         obtaining original installation and diagnostic media, documentation and deliverables

·         variances from the IQ Protocol

IQ summary report

If the results are extensive, then a summary report indicating whether the installation was performed according to the protocol should be written.

The report should be written by a knowledgeable and responsible person who has reviewed all of the actual results.

The report should state:

·         whether or not the installation procedure was followed (as shown by the documentation generated by the person performing the installation)

·         whether the environmental and ergonomic recommendations made by the supplier were met

·         any deviations from the manufacturer’s procedures or recommendations (these should be explained and justified)

Existing systems

For existing systems, any available documentation from the installation period should be gathered for the IQ.   If there is no documentation available, then the IQ should contain this information:

·         hardware and software description and configuration

·         operating parameters

·         environmental controls

·         confirmation that the above items meet the supplier’s /manufacturer’s recommendations

Introduction

The Operational Qualification (OQ) is a test to ensure that each component of a computer system performs as intended within representative or anticipated operating ranges.

It is equivalent to the testing performed by the supplier during the development process (i.e. module and integration testing) and to a system acceptance test at the completion of the system development process.

Note: OQ should also be performed on non-software components. However, that is outside the scope of this guide.

Testing as part of the development process

If the tests performed by the supplier during the development process were satisfactorily performed and documented by the supplier, then the OQ requirement may be satisfied and the OQ may be wavered.

Information about the types of testing should be obtained during:

·         supplier audit

·         Systems Qualification (SQ)

·         Design Qualification (DQ)

The OQ may not be waived for the following systems:

·         operational control systems

·         automated equipment with embedded PLCs which are connected to manufacturing process control and monitoring instrumentation

OQ Protocol

An OQ Protocol should be written.

An OQ Protocol describes the approach used to qualify the system. It should include:

·         execution instructions

·         qualification scripts

·         qualification data and data set-up requirements

·         justification for the choice of data

·         expected results

·         resolution procedure for unexpected results

·         acceptance criteria for qualification - this will be based on the appropriate and corresponding design specifications

Qualification scripts

Qualification scripts, or test plans, describe the step-by-step procedure for qualifying the system. The procedure may be broken down into multiple discrete scripts for ease of use.

The scripts should verify that the system performs as intended against the system specification created during the development process. They should include information about test conditions such as:

·         security

·         screen flow

·         data validation

·         data updates

There should be a direct reference between the test script and the specification against which the testing is being performed.

Qualification data

The data used with the qualification scripts should be identified.

The datasets should include:

·         data that is typical of the data used in a live situation

·         unacceptable data

·         data at the boundaries of the acceptable range

Qualification results

When the OQ scripts are executed, then the results should be recorded, signed and dated by the executor. Screen prints or electronic logs should be retained to verify the results. Automated testing tools may be used where appropriate to record the results.

If expected results and actual results vary, then the discrepancies should be clearly identified and backed up with the records of action taken to resolve them.

Summary reports

If the OQ generates extensive documentation, then a summary report should be written.

This report may be reviewed by the System Validation Steering Team instead of reviewing all the raw data. (Note: You should still retain the raw data.)

The report should be written by a knowledgeable and responsible person who reviews all the raw data.

The summary report should include this information:

·         whether or not the qualification scripts were followed

·         whether or not the expected results were attained

·         description of any deviation from expected results

·         any follow-up activities to correct any deviations

·         statement of whether the operational qualification results meet the acceptance criteria

·         justification for the acceptance of the validation of the system

Existing systems

Any testing or qualification performed on an existing system during its development should be reviewed. This includes a review of the operating history, including problem logs of the system.

If there are a significant number of software-related problems, then a more extensive OQ may be required. If there are few problems, then an OQ may not be required.

Introduction

The Performance Qualification (PQ) ensures that the total system performs as intended in the specified operating range. The total system includes all hardware and software components, associated equipment, people and procedures that make up the system.  The execution process is conducted using company specific pre-defined dataset or actual live data.

PQ is not validation

Performance Qualification is not the same as validation. In earlier literature and in the industry, Performance Qualification was often called validation or validation testing. The two processes should not be confused. Performance Qualification is one process in a series of processes, which make up validation.

During the development process a system acceptance test will have been performed - either at the supplier site or at the user site. This system acceptance test forms part of the PQ.

The PQ should always be performed at the user site (and may involve repetition of all or part of the system acceptance tests as required) and will include testing specific to the user environment and defined ways of working.

PQ Protocol

A PQ Protocol should be written.

The PQ protocol describes how the PQ should be performed. It should contain:

·         description of the use of the system in the context of the work environment

·         references to SOPs or other user documentation, and the user requirements/functional specification as required

·         qualification scripts

·         qualification data

·         justification for the choice of data

·         expected results

·         data set-up requirements

·         testing procedure

·         resolution procedure for unexpected results

·         acceptance criteria for qualification

Qualification scripts

The qualification scripts describe the procedures to verify the performance of the system against the User Requirements Specification. They should simulate the operation of the system in a live situation, using all system components and operating procedures.

Scripts may be broken down into multiple steps for ease of use.

There should be a direct reference between the test script and the specification against which the testing is being performed.

Setting operational capacity limits

The PQ should also include testing the system against (but not exceeding) its operational capacity.

Note: If the system is expanded to operate at a higher capacity this type of testing should be repeated.

The operational capacity should be set by the user but should not exceed the rated capacity provided by the supplier.

Qualification data

The data to be used with qualification scripts must be identified. The choice of data should be justified in terms of its suitability for demonstration purposes. It should simulate the data used in live situations.

Abnormal data or data which is outside the operating ranges should also be tested to ensure that it is handled correctly in the system.

If a new computer system is to be implemented as part of a computer-controlled process in a manufacturing environment, process validation requirements should be considered.

PQ results

When the scripts are executed, then the executor should record, sign and date the results. Screen prints or electronic logs should be retained to verify the results. Where appropriate, automated testing tools may be used to record results.

If discrepancies between expected and actual results are identified, then they should be resolved. Action taken to resolve discrepancies should be documented.

Summary report

If the PQ generates extensive documentation, then a summary report should be written.

This report may be reviewed by the System Validation Steering Team instead of reviewing all the raw data. (Note: You should still retain the raw data.)

The report should be written by a knowledgeable and responsible person who reviews all the raw data.

The summary report should include this information:

·         whether or not the qualification scripts were followed

·         whether or not the expected results were attained

·         description of any deviation from expected results

·         any follow-up activities to correct any deviations

·         statement of whether the performance qualification results meet the acceptance criteria

·         justification for the acceptance of the validation of the system

Existing systems

A PQ Protocol should be developed for an existing system in the same way as for a new system. The PQ should cover all of the functions outlined in the user requirements specification.

Historical information may be used in lieu of performance qualification scripts, data and expected results. However, the actions taken, the data used and the results obtained when the historical data was generated must be clear.

In addition, effective change control must have been in place to ensure that the system has not changed since the historical data was generated.

If there is not enough data for some or all of the functions, then the gaps must be qualified as for a new system. The protocol must specify which system functions are qualified on historical information and which ones will be qualified as new.