1

WEBINAR

How to Validate Computerized GxP Systems:

Is your documentation accurate, comprehensive and
accessible?

Presented by Chrysa Plagiannos

Senior Validation & Compliance Analyst
Montrium
© Montrium Inc. 2016
 Your Presenter
2

Chrysa Plagiannos
Senior Validation & Compliance Analyst – Montrium

Educational Background:
• Degree in Chemical Engineering from McGill University
Career Overview:
• 13 years experience in pharmaceutical industry
• Extensive validation experience, including validation of manufacturing processes,
equipment, facilities, and computerized systems
• Experience in QA functions (approval of controlled documents, complaint
management)
• Experience in Engineering functions (equipment selection and purchasing,
investigation reporting)

© Montrium Inc. 2016

 About Montrium
3

• Montrium is a technology company focused on the delivery of enterprise content

management solutions and professional services for the Life Sciences industry.
• Montrium’s ECM platform, Montrium Connect, consists of solutions for Electronic Trial
Master Files, Regulatory Document Management & Submission Planning and Quality
Management.
• We offer also services in validation, quality assurance, auditing and strategy
surrounding technologies.
• Montrium was founded in 2005 and is headquartered in Montreal.
• Montrium works with pharmaceutical companies in North America, Europe and Asia.

© Montrium Inc. 2016

 Today’s Focus
4

1. Introduction 3. Executing Test Scripts

• What is a GxP System? • Setting up Prerequisites
• What is an Electronic Record? • Good Documentation Practices
• Why is Testing Important? • Creating Screen Captures
• Validation Terminology • Documenting Non-Conformances
2. Preparation for Testing 4. Traceability & Reporting
• Planning 5. Conclusion & Recommendations
• Defining the scope of testing
• Tips for writing test scripts
• Assembling a Team

© Montrium Inc. 2016

 5

1
Introduction
• What is a GxP System?
• What is an Electronic Record?
• Why is Testing Important?
• Validation Terminology

© Montrium Inc. 2016

 What is a GxP System?
6

• GxP: Set of compliance regulations including but not limited to, Good Laboratory
Practice (GLP), Good Manufacturing Practice (GMP), and Good Clinical Practice
(GCP).

Examples of computerized systems used for GxP activities :

- Systems used to generate electronic records (ex. training records,

complaint files, change requests, incident reports).

- Systems used to store/ manage data that is subject to regulations, such

as adverse event reports and raw data obtained during laboratory and
production activities

© Montrium Inc. 2016

 What is an Electronic Record?
7

Definition as per 21 CFR Part 11.3:

“Electronic record means any combination of text, graphics, data, audio, pictorial, or
other information representation in digital form that is created, modified, maintained,
archived, retrieved, or distributed by a computer system.”

© Montrium Inc. 2016

 Why is Testing Important?
8

Reason 1: Validation provides documented evidence that a system is fit for its
intended use.
• System requirements provide an objective standard to which the system is tested
during validation.
• System functionality is tested by executing test scripts designed to demonstrate
that the requirements were met.
• Requirements are based on:
– Regulatory requirements
– End-user/ business needs
– Risk assessment/ mitigation

© Montrium Inc. 2016

 Why is Testing Important?
9

Reason 2: Validation documents are auditable records.

• Inspectors are looking for:
– Documented evidence that validation activities were performed in accordance with approved
procedures
– Readily accessible records (can be produced in a timely manner)
– Proof of sufficient testing, based on intended use and potential risk
• Scope of validation should be tied to the risk the system poses to:
– Patient safety
– Product quality
– Data integrity

© Montrium Inc. 2016

 Why is Testing Important?
10

Reason 3: Electronic records managed by GxP systems can be auditable

records.
• These records will also be presented to Inspectors/ Auditors
• Examples of records
– Complaint files
– Certificates of Analysis
– Standard Operating Procedures (SOP)
• Failure to perform adequate testing could lead to:
– Inability to produce documents upon request
– Loss of critical data
– Compliance violations ex. missed regulatory filing deadline

© Montrium Inc. 2016

 Why is Testing Important?
11

• Real-life Example: Warning Letter

© Montrium Inc. 2016

 Why is Testing Important?
12

• Summary of Issues Identified in the Warning Letter:

– System not fully validated upon release
– Critical issues found with interim report used to release system:
• Inadequate training
• Incomplete documentation (SOPs and work instructions)
• Inaccurate data migration of legacy database
– Technical issues with system also identified:
• Inaccurate dates captured in system
• Follow-up information recorded as new event
– System deficiencies led to inaccurate assessments of adverse events and untimely submission
of alerts

© Montrium Inc. 2016

 Validation Terminology
13

• Qualification: Process of demonstrating whether an entity is capable of fulfilling specified

requirements.

• Validation: Process of establishing documented evidence which provides a high degree

of assurance that a specific process will consistently produce a product meeting its
predetermined specifications and quality attributes

• Note: Qualification is part of validation.

• Visit Montrium blog for more information

© Montrium Inc. 2016

 Validation Terminology
14

© Montrium Inc. 2016

 Types of Testing
15

• Verification that the appropriate system was delivered and that it is installed correctly (i.e. in a
manner consistent with requirements).
• Ex. IQ
• Verification that the system is capable of consistently operating in accordance with established
functional specifications.
• Ex. OQ
• Verification that the system consistently operates in accordance with established requirements in
its typical operating environment and is fit for its intended use.
• Ex. UAT, PQ
• Note: Terminology can vary from organization to organization. What’s important is to ensure that
the system is thoroughly tested regardless of the term used to describe the testing.

© Montrium Inc. 2016

 16

2 Preparation for Testing

•
•
•
Planning
Defining the scope of testing
Tips for writing test scripts
• Assembling a Team

© Montrium Inc. 2016

 Validation Planning
17

• Validation Plans should be formally documented

• Types of Planning Documents:
– Validation Master Plan
– Qualification/ Validation Plan(s)

Stage 2:
Application (s) Verified by Validation Plan(s) for
Application
Governed by
Validation Master
Plan
Infrastructure Stage 1:
• Physical Hardware Verified by Infrastructure
• Virtual Machines Qualification Plan

GxP System
© Montrium Inc. 2016
 Validation Planning
18

• Information to be included in the Planning Document(s):

– Scope of qualification/ validation activities
– Project roles and responsibilities
– Required deliverables
– Required procedural controls

• For simple systems with few requirements, validation deliverables can be combined (ex. combined
IQOQ Protocol).

• Important: Conduct validation activities in accordance with the Plan. If the sequence of activities
outlined in the plan is not followed, document any outstanding issues along with the justification
for proceeding.

© Montrium Inc. 2016

 Scope Definition
19

• Ensure that the scope is clearly defined.

• For software, it is important to understand the system’s core functionality to be

able to define the scope of testing

© Montrium Inc. 2016

 Scope Definition (GAMP®5)
20

• GAMP®5 methodology can be used to define the scope

• What is GAMP®5?
– Guidance documentation which provides a risk-based approach to computer system
validation
– A system is assigned to a category based on its intended use and complexity.
• Determine the system Software Category as per GAMP®5
• This category is based on the extent to which the system is customized since the
risk of failure and defects generally increases as the complexity of the system
increases
• Rule of Thumb: Increased customization leads to more extensive testing

© Montrium Inc. 2016

 Where to Test
21

• Testing is often performed in multiple environments

• It is recommended to perform testing in three different environments:
– Development: Serves as a non-controlled, pre-QA environment for informal testing
– Test (QA): Serves as a controlled, pre-Production environment for testing and training
– Production: Serves as a controlled operational environment for day-to-day activities
• It is not necessary to perform the same tests in all three environments. Where
testing is done is determined using a risk-based approach

© Montrium Inc. 2016

 Advantages of Testing in Multiple Environments
22

• Prevent the transfer of issues to Production environment

• Avoid creating dummy data within the Production environment
• Testing that may be destructive to the Production environment can be performed
elsewhere (ex. infrastructure stress testing)
• Reduced system downtime
• Reduced maintenance and support costs

© Montrium Inc. 2016

 Test Scripts: Basic Characteristics
23

• Test Script: Used to define and document testing activity

• Traceable to requirement(s).
• If the test passes, the associated requirements have been met.
• Main Test Script Components:
– Test Objective
– Acceptance Criteria
– Prerequisites
– Test Instructions (Step-by-Step)
– Expected Results
– Designated spaces for recording what was observed

© Montrium Inc. 2016

 Test Scripts: Recording Results
24

• Data recorded by testers:

– Actual Results
– Identification of testing sequence (ex. run number)
– Overall Result (i.e. Pass, Fail, Conditional Pass, Not Performed)
– Cross-references to attachments
– References to any non-conformances
– Tester identification information (ex. name/ initials)
– Date actions were performed

© Montrium Inc. 2016

 Example: Test Script
25

Run
Number

Actual
Result Tester
Name
and Date

Reference to
Prerequisites

Step by Overall
Step Reference Result
Instructions supporting
docs

© Montrium Inc. 2016

 Test Scripts: Recording Results
26

It is recommended that Actual Results be written in full i.e. document exactly what was observed,
instead of simply writing “As Expected”
• Why? : Testers are more likely to detect/ report issues when additional details are
provided.

Poll: How does your organization record actual

results during test script execution?

Option 1: Actual results are written in full

or
Option 2: Results are documented as ‘as expected’
or ‘not as expected’

© Montrium Inc. 2016

 27
Characteristics of Well-Written Test Scripts

• Clearly defined prerequisites

• Concise, reproducible test instructions

• Fields available for recording data

• Precise expected results

• Results independently review

© Montrium Inc. 2016

 How to Record Results? 28

Electronic, Paper or Hybrid

• Execution can be performed in the following ways:

– Electronically: Results are recorded electronically, use of e-signatures
– On Paper: All results documented by hand
– Hybrid Approach: Results documented electronically, printed and signed off
with wet-ink signature
• Executing electronically means that the executed test script is
considered an Electronic Record
• Electronic execution process must be validated to ensure that the
solution is fit for use and compliant with applicable regulations

© Montrium Inc. 2016

 Advantages to Executing Test Scripts Electronically 30

• Managing documents electronically

– Trace tests electronically to requirements
• Accessibility of Records
– Remote access to documentation
– Can lead to reduced approval times
• Secure Access
– Can assign permissions to electronic documents
• Save Time
– Easier to fill out (type results, use of drop-down menus)
• Save Money
– No costs related to printing, scanning and storage of physical documents

© Montrium Inc. 2016

 Review of Test Results
31

• Test results are independently reviewed (by a member of the validation team
other than the tester).
• Provides assurance that execution was done properly and facilitates the reporting
of any issues.
• This review involves:
– Ensuring that executed test script is complete
– Verifying that all appropriate documentation has been generated
– Checking cross-references to supporting documents
– Confirming the overall result

© Montrium Inc. 2016

 Time to Assemble Your Testing Team
32

• Choose testers: Review the testing

documentation and determine how many
people are required for testing
• A good tester should:
– Understand validation and quality principles
– Be knowledgeable of good documentation
practices
– Be familiar with the system being tested
– Be aware of the purpose of testing
• Select testers based on the type of testing
being performed

© Montrium Inc. 2016

 Who Should You Recruit?
33

Type of Testing Who should be involved? Advantages

Technical Testing Involve technical personnel: • Leverage system
(ex. IQ) • System Administrators knowledge
• IT Professionals • Subject Matter
Experts (SME) to
evaluate issues

Business Process Involve end-users: • Ability to assess

Testing (ex. UAT) • Training Managers system suitability
• Investigators • Training opportunity
• QA

© Montrium Inc. 2016

 Train Your Testing Team
34

• Ensure testers have been adequately trained prior to participating in validation

activities
• Especially important when involving non-validation personnel (i.e. IT, end-users)
• Conduct training in accordance with approved procedures
• Successful completion of training must be documented
• There are 2 levels of training:
– System Training
• Ex. Training on how to use the system
– Validation Training
• Ex. Training on validation policy/ procedures, documenting non-conformance, good documentation
practices etc.

© Montrium Inc. 2016

 36

3
Executing Test Scripts
• Setting up Prerequisites
• Good Documentation Practices
• Creating Screen Captures
• Documenting Non-Conformances

© Montrium Inc. 2016

 Preparing Prerequisites
37

• Prerequisites: Conditions that must be satisfied prior to executing a test script

• Recorded within the executed test script document
• Examples of Prerequisites:
– Definition of technical components, such as server names, IP addresses
– Presence of data that allows for a particular scenario to be tested
• Tips:
– Gather supporting documentation, such as user guides
– Create dummy data before starting to test

© Montrium Inc. 2016

 Example of Prerequisites
38

© Montrium Inc. 2016

 39
Good Documentation Practices

• Governed by internal procedures.

• Ensure that testers have been trained prior to execution.
• Principles of Good Documentation Practices:
– Accurate
– Legible
– Contemporaneous
– Original
– Attributable

© Montrium Inc. 2016

 Annotations: Correcting Text
40

• Any correction to preprinted text may be made only if it does not alter the
original intent of the test.
• Corrections to preprinted or handwritten text may be made by crossing out the
incorrect information with a single line stroke and without obscuring the original
information.
• The correct information may be entered below, above, or next to the error.
• If additional space is required, use a numbered footnote and record the correct
information in the page margin.
• All corrections must be initialed and dated

© Montrium Inc. 2016

 Annotations: What Not to Do
41

Correction Error in Annotation

This result is bad. good Correction not initialed/ dated.
This result is bad. good Original text is obscured.
CP 15-JUN-2016

This result is bad. No correction was made.

CP 15-JUN-2016
This result is bad. Good Illegible correction.
CP 15-JUN-2016

© Montrium Inc. 2016

 Annotations: Best Practices
42

Correction
This result is bad. good 1

This result is bad. good CP 15-JUN-2016

1 Entry Error. CP 15-JUN-2016

© Montrium Inc. 2016

 When is an Annotation Allowed?
43

• When there is no impact on the original intent of the test

Test Execution Instructions & Results

Step Initial/
Step Instructions Expected Result Actual Result
Status Date
12. Verify that the Training Report The Training Result:
displays training requirements. Report displays
a) Navigate to the Training training
______________
Report. requirements.
Attachment:
b) Verify that training
requirements are displayed
for the Trainee identified in ______________
prerequisite PRQ-1. PRQ-2 Validation Non-
CP 15-JUN-2016 Conformance:

© Montrium Inc. 2016

 When Are Annotations Not Allowed?
44

• When a correction would alter the original intent of the test

• Should be addressed via a non-conformance report
Test Execution Instructions & Results
Step Initial/
Step Instructions Expected Result Actual Result
Status Date
13. Verify that a task is assigned to the A task is assigned Result:
Trainee. to the Training
a) From the Training Manager.
______________
Management application,
navigate to the Task List. Attachment:
b) Verify that a new task was
assigned and that it is _______________
associated to the Training Validation Non-
Course identified in Conformance:
prerequisite PRQ-5.
c) Generate a screen capture.

© Montrium Inc. 2016

 Why Are Screen Captures Important?
45

• Screen captures provide evidence of what was observed in the system during
testing (typically included in attachments)

• Used to corroborate the testers’ findings i.e. show whether the system’s state
matches the expected result

• While not mandatory, they facilitate review by third party (QA, Auditor/ Inspector)

© Montrium Inc. 2016

 When are Screen Captures Necessary?
46

• Proving Step: Demonstrates that the test objective was met

• Non-Proving Step: Intermediary step that allows the tester to perform a proving step (no screen capture required)

© Montrium Inc. 2016

 Tips for Generating Screen Captures
47

• Make sure screen captures depict the elements being verified

• Take screen captures only when necessary (proving vs non-proving steps)
• Ensure that screen captures are legible especially when printed
• When possible, screen captures should include test conditions which prove that
the test was properly executed (ex. URL, login names)

© Montrium Inc. 2016

 Screen Captures: What Not To Do
48

• Verification: Email attachment of Non-Conformance document was received by

Tester3 QA

• Issues with this Screen Capture:

– Does not show email recipient name, i.e. Tester3 QA
– Attachment information is not visible

© Montrium Inc. 2016

 Screen Captures: What Not To Do
49

• Verification: Email attachment of Non-Conformance document was received by

Tester3 QA

• Issues with this Screen Capture:

– Relevant information is not visible
– Image distorted by resizing (illegible)
– Image of entire screen is not necessary in this case

© Montrium Inc. 2016

 Screen Captures: Best Practices
50

• Verification: Email attachment of non-conformance document was received by

Tester3 QA

• Why is this an appropriate screen capture?

– Relevant information is visible
– Use of highlight to draw attention to elements verified (if your procedures permit the use of
highlight)
– Image is not distorted
© Montrium Inc. 2016
 What are Non-Conformances?
51

• Non-conformances: Event that prevents test acceptance criteria from being met.
• Terminology can vary. Other commonly used terms are:
– Deviation
– Exception
– Test Incident
• Examples of non-conformances:
– Discrepancy between expected and actual result
– Errors that occurred during execution
– Changes to test methodology resulting in test’s intent being altered

© Montrium Inc. 2016

 Poll Question:
52

What does your organization call non-conformances?

• Non-Conformances
• Deviations
• Exceptions
• Test Incidents
• Other

© Montrium Inc. 2016

 53
Documenting Non-Conformances

• A non-conformance report should:

– Be clear and concise
– Assess impact (blocking vs non-blocking issue)
– Pinpoint root cause(s)
– Incorporate SME input
– Propose appropriate corrective action(s)
– Provide a rationale for actions taken
– Avoid overly technical language, define all abbreviations
– Include supporting documentation when necessary

© Montrium Inc. 2016

 Resolving Non-Conformances (Step-by-Step Approach)
54

• Step 1: Provide a detailed description of the non-conformance

• Step 2: Determine the root cause of the issue reported
– Examples of possible root causes:
• System configuration error
• System deficiency
• Execution error
• Protocol generation error
• Step 3: Identify corrective actions (approved prior to implementation)
• Step 4: Implement corrective actions and provide documented evidence that the
issue is resolved

© Montrium Inc. 2016

 Example: Non-Conformance Description
55

© Montrium Inc. 2016

 Example: Non-Conformance Investigation
56

© Montrium Inc. 2016

 Example: Non-Conformance Corrective Action/ 57

QA Approval

© Montrium Inc. 2016

 58

4 Traceability and Reporting

© Montrium Inc. 2016

 Traceability
59

• Provide a link between system requirements and the test scripts/procedural controls that
verify them.
• Reminder: System functionality is tested by executing test scripts designed to
demonstrate that the requirements were met.
• Establishing traceability provides proof that the system functions as intended.
• May be presented in a Traceability Matrix or for simple systems (few requirements) can be
documented within another validation deliverable.
• A requirement may not be met via testing alone. Procedural controls are often necessary.
• If a requirement is not explicitly tested, a rationale should be provided (risk-based
approach).

© Montrium Inc. 2016

 Example: Traceability Matrix
60

© Montrium Inc. 2016

 Summary Report
61

• Issued at the end of the validation.

• Main components:
– Identification of all deliverables (Document ID, approval date)
– Test Result Summary
– Discussion of any non-conformances encountered during testing
– Statement of system acceptance (specify any system limitations)
• Report is approved by stakeholders.
• Possible stakeholders (depends on type of validation done):
– Process Owner
– System Owner
– QA

© Montrium Inc. 2016

 Conclusions and Recommendations
62

• Proper planning is key

• Team Work
– Involve stakeholders
– Pick the right people for each phase of the project
• Well-written test scripts lead to smoother test executions
• Document any testing issues via non-conformance reports
• Be sure to establish traceability
• Prepare a Summary Report (often reviewed by Inspectors)
• If it’s not documented, it didn’t happen!

© Montrium Inc. 2016

 63

Thank you for your attention!

Questions

?
cplagiannos@montrium.com

© Montrium Inc. 2016

 64

Have a question? Get in touch!

Schedule a call with one of our validation experts

Montrium Inc. Montrium S.A.

507 Place d’Armes, Suite 1050 2A, rue Alophe Diederich
Montreal (QC) L-5820 Fentange
H2Y 2W8 Luxembourg
Canada +352.20.88.01.30
+1 (514) 223-9153

info@montrium.com
www.montrium.com

© Montrium Inc. 2016