QCC ITSV 001 001 IT Software Validation[1]

IT SOFTWARE VALIDATION DOCUMENT TYPE :

REF. CODE:

ISSUE NO:

ISSUE DATE:

GUIDANCE

QCC-ITSV-001

001

6 DECEMBER 2007

Index 1. Introduction ............................................. .................................................................... ............................................... .............................................. ........................ 1 2. Aim .............................................. ..................................................................... .............................................. .............................................. .................................. ........... 2 3. Scope .............................................. ..................................................................... .............................................. .............................................. ............................... ........ 2 4. Definitions ............................................ ................................................................... .............................................. ................................................ ......................... 2 5. Software categories ............................................... ....................................................................... ................................................ .............................. ...... 3 5.1 Commercial off-the-shelf (COTS) software ..................................... .................................................. ............. 3 5.2 Modified off-the-shelf (MOTS) software ...................................................... ...................................................... 3 5.3 Custom software software (self-developed software) .............................................. .................................................. .... 4 6. Software life cycle ............................................ ................................................................... ............................................... .................................... ............ 4 7. Validation .............................................. ..................................................................... .............................................. ............................................... ........................ 5 7.1 Validation diagram .............................................. ...................................................................... .......................................... .................. 5 7.2 Software checklist ............................................... ....................................................................... .......................................... .................. 6 7.3 Validation report .............................................................. ...................................................................................... .............................. ...... 6 7.4 Good practices ........................................... .................................................................. .............................................. ............................. ...... 7 8. Configuration management .............................................. ...................................................................... ........................................... ................... 7 9. References ............................................. .................................................................... .............................................. ............................................... ........................ 8 Appendix 1 ............................................ ................................................................... ............................................... ............................................... ........................... .... 9 Appendix 2 ............................................ ................................................................... ............................................... ............................................... ......................... .. 1 3

1.

Introduction

This document presents general validation principles that QCC considers applicable to the validation of software used in a forensic laboratory. It is based on generally recognized software validation principles and could therefore be applicable to any software. Validation is an activity that answers the question “Is the software suitable for its intended purpose?” Provided purpose?” Provided the verification activity has been properly performed, that question translates roughly to “Does the software meet its requirements when it's run?” [11]. It is recommended that validation activities be conducted throughout the entire software life cycle. The level of validation effort to be applied and validation techniques to be used must be determined for each software project.

Ref code: QCC-ITSV-001

Issue No. 001

Page: 1/17

2.

Aim

The aim of this document is to provide general information on validation of software used in accredited laboratories. Because detailed papers on software validation have been already published, the present document provides references to some papers that might be useful when validating self developed software.

3.

Scope

The scope of this document is validation of all types of software used in forensic laboratories.

4.

Definitions

Computer system A group of hardware components and associated software designed and assembled to perform a specific function or group of functions. Software A collection of programs, routines, and subroutines that controls the operation of a computer or a computerized system. Software product The set of computer programs, procedures and associated documentation and data. Software item Any identifiable part of software product. Testing The process of exercising or evaluating a system component by manual or automated means to verify that it satisfies requirements or to identify differences between expected and actual results. Verification Confirming that the output from a development phase meets the input requirements for that phase. Validation Establishing by objective evidence that all software requirements have been implemented correctly and completely and are traceable to system requirements. Life cycle model A framework containing the process, activities, and tasks involved in the development and maintenance of a software product, spanning the life of the software from the definition of its requirements to the termination of its use, i.e. from concept to retirement.


Issue No. 001

Page: 2/17

5.

Software categories

The software products in computers and automated systems used for the acquisition, processing, recording, reporting, storage, or retrieval of data can be classified into 3 categories, according to the amount of work required ensuring validation. •

COTS – Commercial off-the-shelf software

•

MOTS – Modified off-the-shelf software

•

CUSTOM - Self-developed software

These three types of software can reside on local hard drives, network hard drives, embedded on integrated circuits (IC, ROM, EPROM), or removable disks. 5.1 Commercial off-the-shelf (COTS) software Commercial off-the-shelf software is most commonly used in forensic laboratories. It is a code that is purchased without modification and either, cannot, or will not, be modified by the laboratory and is used “as is”. Almost all software bought by the average computer user fits into the COTS category: operating systems, office product suites, word processing, and e-mail programs etc. An example of this would be Microsoft Office or dedicated instrument interface. This software can be considered adequate by the “mass market” and used as is. The laboratory should have a set of requirements that describe what the software should do and there should be testing to ensure that the software meets these requirements. Embedded IC software inside instruments or automated equipment can be regarded as validated by the manufacturer and confirmed as adequate during the calibration process. This assumes the calibration fully exercises the firmware functions. No further action is necessary. Off-the-shelf operating systems need not be validated as separate programs. However, system-level validation testing of the application software should address all the operating system services used, including maximum loading conditions, file operations, handling of system error conditions and memory constraints that may be applicable to the intended use of the application program. There should be evidence that any formula strings written by the laboratory work as expected. Any loadable parameters should be considered as data transfer and be “checked”. 5.2 Modified off-the-shelf (MOTS) software A MOTS (either modified or modifiable off-the-shelf) product is typically a COTS product whose source code can be modified or customised for specific applications by the purchaser, by the vendor, or by another party to meet the requirements of the customer. The purchased portion of the software can be considered COTS. The modified or customised portion is considered CUSTOM and should have evidence of validation. This can be achieved by documenting the functions and design of the modification. The code or blocks of the modification should be both documented and annotated showing eac h function. Finally, testing should be performed on each block of the modification and evidence given that each piece works as designed and satisfies each function.


Issue No. 001

Page: 3/17

5.3 CUSTOM software (self-developed software) Custom software is a code that is laboratory written or subcontractor written by means of some commercial standard or configurable software package. Examples of this include spreadsheets, applications written in MS Visual Basic or C++, SQL database design, etc. Software products self-developed by the laboratory require full validation. The software packages themselves do not require validation, but new versions should be always treated with caution and should be tested and approved before use. Beta-releases should never be used. It should especially be noted that spreadsheets are programs, and that they as such require validation. Spreadsheets may be validated as other programs. Special attention should be paid to the fact that spreadsheets have a wide-open user interface what makes them very vulnerable to unintentional changes.

6.

Software life cycle

All software has an activity lifecycle. Activities in a typical software life cycle include: •

Management

•

Requirements

•

Design

•

Implementation (Coding)

•

Integration and Test

•

Installation

•

Operation and Support

•

Maintenance

Following this lifecycle assures quality and provides validation evidence. Minimally, there should be evidence of requirements, design, implementation, testing, and installation [10]: •

Requirements – Defining, in user terms the functions of the software (what the software needs to do). The requirements should be included in a Functional Requirements document.

•

Design – Design of how the software is built, along with any diagrams illustrating the architecture. The output is a Software Design document.

•

Implementation/Coding – Writing the actual source code including header information and annotations. The code is the document.

•

Testing – Software testing according to a test plan of what will be tested and how, with individual test cases, along with successful completion, which satisfy the plan, and map directly back to the requirements. Important points to consider are verification of math, valid/invalid inputs, power failure, and code walkthrough’s 1, structural and functional tests.

•

Installation/Checkout – Loading the software onto laboratory hardware.

1

A walkthrough is a term describing the consideration of a process at an abstract level. The term is often employed in the software industry to describe the process of inspecting algorithms and source code by following paths through the algorithms or code as determined by input conditions and choices made along the way. The purpose of such code walkthroughs is generally to provide assurance of the fitness for purpose of the algorithm or code; and occasionally to assess the competence or output of an individual or team. Something akin to walkthroughs are used in very many forms of human endeavour since the process is a thought experiment that seeks to determine the likely outcome(s) of an affair based on starting conditions and the effects of decisions taken (http://www.encyclopedia.laborlawtalk.com). Ref code: QCC-ITSV-001

Issue No. 001

Page: 4/17

These processes have to be performed, and documentation updated, each time there is a change to the software - this is the reason for calling the process a “software life cycle”. The validation documentation need not be too cumbersome. One document would suffice for simple software, and it would be separated into several documents for more complex projects.

7.

Validation

Software validation should not be left to the end of a project; in fact quite the reverse is required. The validation should start at the beginning of a project as it is shown in the diagram below. 7.1 Validation diagram The following diagram, published by Soft Solution Internati onal, shows bad versus good practice in software validation.


Issue No. 001

Page: 5/17

7.2 Software checklist The following is a good checklist to verify that adequate validation activities have occurred [10]: 1. Has the firmware been validated, via calibration by a cal lab, which has the capability to thoroughly check the software (i.e. OEM or authorised partner)? (Note: for noncalibratable device firmware, treat as software) 2. Have all of the “used” firmware parameters been documented and confirmed as correct? 3. Does a Software Requirements document exist for the software? 4. Does a Software Design document exist detailing either the full design or details of the configuration? 5. Do software testing documents exist describing completed, unique, test cases that exercise the design both +/- and confirm the requirements? 6. Is there a test log showing test failures and corresponding retests/dispositions? 7. Does evidence exist confirming correct software deployment at each target installation? 8. Has configuration control been applied to all of their software/firmware to ensure that: a) The software source code location is access controlled. b) Firmware/software formulae & parameters are “locked” to prevent inadvertent changes. c) Equipment lists identify software as a separate line item showing correct version and location. 9. Does evidence exist showing that personnel involved in custom software development have adequate training? 10. Do Databases and spreadsheets include “audit trails” to avoid previous data being obscured? 11. Do adequate instructions exist for the operation & maintenance of the software? 12. Does the accuracy of the firmware/software meet or exceed the accuracy required by the test method? 7.3 Validation report A software validation report should consist of 5 sections: 1. Objectives and scope of application including (description of the software product, a list of the involved persons, and specification of the type of software to determine the extent of the validation); 2. Software life cycle overview (specification of date and signature for the tasks of preliminary work and the peer reviews assigned to each life cycle phase); 3. Software life cycle activities (information that is relevant for the validation, having all topics outlined, it should be easier to write the report); 4. Conclusion. (for the persons responsible to conclude and sign the validation report); 5. References and annexes. A template of such report has been published by Nordtest Software [Method of Software Validation, NT Techn. Report 535 ]


Issue No. 001

Page: 6/17

7.4 Good practices The following are some good software practices that have been found to assist in managing validation [10]: •

•

treat each software product as a piece of calibration equipment that has to be “recalibrated” each time it is changed or modified; place software product masters in a read-only directory;

•

network computers that access a shared program on a server;

•

lock spreadsheet cells that contain math;

•

password protect configuration files or setup screens;

•

back up the software;

•

plan for hardware/software disaster recovery.

8.

Configuration management

Configuration management ensures that all changes to software/hardware are controlled. It also ensures that all software installations are known, and have periodic checks. To achieve this, software used in accredited laboratories should be considered with regard to where it fits into the computing hardware. A software product can be installed on many computer systems. Many laboratories have one version of software product that is installed on many computer systems. It is important to separately consider each installation. In all cases, the software should be under, software configuration management, version and access control. The labs should have records indicating what versions are current. They should also know which computer systems have what software products installed. They should also control access to them so that only authorised individuals have access. There should be a “check” that is performed periodically to ensure that the correct version is installed and no unauthorised modifications have occurred [10].


Issue No. 001

Page: 7/17

9.

References

1. ISO/IEC 17025 General Requirements for the Competence of Testing and Calibration Laboratories, 1999 2. ISO/IEC/IEA/IEEE 12207 – 1995 “Information Technology – Software Lifecycle Processes” 3. ISO/IEC 12119 – 1994 “Information Technology – Software Packages – Quality Requirements and Testing” 4. ISO9000-3 “Guidelines for the application of ISO9001 to the development, supply, and maintenance of software” 5. ASTM E919-96 “Standard Specification for Software Documentation for a Computerized System” 6. ASTM E 1579 - 93 : “Standard Guide for Ensuring Data Integrity in Highly Computerized Laboratory Operations” 7. ANSI/IEEE Std 1012-1998 “IEEE Standard for Software Verification and Validation Plans” 8. ANSI/IEEE Std 1059-1993 “IEEE Guide for Software Verification and Validation Plans” 9. EPA 2185 – 1995 “Good Automated Laboratory Practices 10. Gregory D. Gogates , Software Validation in Accredited Laboratories. A Practical Guide. http://www.a2la.org/guidance/adequate_for_use.pdf ftp://ftp.fasor.com/pub/iso25/validation/adequate_for_use.pdf, 11. Chuck Taylor , Some Notes on Independent Verification, Validation, and Accreditation of Computer Software. 12. A96 CAEAL (Canadian Association For Environmental Analytical Laboratories) Policy on the use of Computers in Accredited Laboratories, Rev 1.2, May 2005, http://www.caeal.ca/A96-CAEAL%20Lab%20IT%20Pol.pdf


Issue No. 001

Page: 8/17

APPENDIX 1 COMMON REFERENCES WITHIN ISO/IEC 17025 THAT APPLY TO THE USE OF ELECTRONIC SYSTEMS IN AN ACCREDITED LABORATORY [12]. Clause Extract / Wording 4.1.5.c “…shall have policies and procedures to ensure the protection of its clients’ confidential information and proprietary rights, including procedures for protecting the electronic storage and transmission of results...” 4.3.1 “…shall establish and maintain procedures to control all documents…. ….. in this context, “document” could be … …. software…. These may be on various media, whether hard copy or electronic, …..” 4.3.2.1 “All documents issued… ….shall be… …reviewed and approved for use….”

Policy Consideration Integrity of data and Access control Procedures exist to protect client’s information

Integrity of data and Access control Procedures to control software

Integrity of data Quality system reviewed and approved by authorized personnel by electronic signatures or password protection and/or retention of approval records in hard copy. 4.3.2.2 “The procedure(s) adopted shall ensure Integrity of data and Retrieval of that: data a) authorized editions of appropriate Authorized editions of appropriate documents are available at all documents all locations. (Intranet, locations…..” NT file Share) 4.3.3.2 “..the altered or new text shall be Integrity of data identified…” Altered or new text shall be identified (electronic document) 4.3.3.4 “Procedures shall be established….. Integrity of data …documents maintained in Procedures shall describe how computerized systems are made and changes in documents, including controlled”. software are controlled. 4.12.1.2 “All records… …shall be… …readily Retrieval of data retrievable…” “…hard copy or Records (electronic media) shall be electronic media…” stored and maintained so that they are retrievable 4.12.1.4 “The laboratory shall have procedures to Integrity of data and Access protect and backup records stored control electronically and to prevent Procedures to protect and back-up unauthorized access to or amendment of electronic records. these records.” 4.12.2.1 “…shall retain records… …to establish Integrity of data and Retrieval of an audit trail…” data Retain records for the retention period (old versions of software also)


Issue No. 001

Page: 9/17

Clause Extract / Wording 4.12.2.2 “Observations, data and calculations shall be recorded…”

Policy Consideration Integrity of data Observations shall be recorded at the time they are made. (electronic). 4.12.2.3 “When mistakes occur in records Integrity of data and Access ,………. In the case of records stored control electronically, equivalent measures shall Electronic records shall avoid loss to be taken to avoid loss or change of original data (audit trails) Do original data”. Databases and spreadsheets include "audit trails" to not allow previously recorded data to be obscured? 5.2.1 “The laboratory management shall Validation and Maintenance of ensure the competence of all who electronic system operate specific…..” Does evidence show that personnel involved in use of Custom Software have adequate training? If the Custom Software was developed inhouse, is there evidence that they have adequate training in the development of these types of solutions? 5.3.4 Access to and use of areas affecting… Integrity of data and Access …quality… …shall be controlled. control Server rooms or server access should have limited access 5.4.1 “The laboratory shall have instructions Integrity of data Validation and on the use and operation of Maintenance of electronic system equipment….” This includes software. Do adequate instructions exist for the operation & maintenance of the software? 5.4.7.1 “Calculations and data transfers shall be Integrity of data and Validation of subject to appropriate checks in a Electronic system systematic manner.” Calculations (spreadsheet) and data transfers (tables) shall be subject to checks. Where other programming approaches are used to effect data manipulation and transfer, there must be some method established to ensure that these are checked as well. 5.4.7.2 “computer software developed by the Validation of Electronic system a) user is documented in sufficient detail Software shall be validated and and suitably validated …..” documented – even if commercial software is configured for specific use 5.4.7.2 “procedures are established for Integrity of data and Access b) protecting data, such procedures shall control include integrity, confidentiality…” Procedures are established to protect data


Issue No. 001

Page: 10/17

Clause 5.4.7.2 c)

5.4.7.2

5.5.2

5.5.4

5.5.5

Extract / Wording “computers and automated equipment are maintained…”

Policy Consideration Integrity of data and Maintenance of Electronic system Computer and automated equipment are maintained Note Validation of Electronic system “Commercial off-the-shelf software… The software validation note allows …in general use, within their design labs to take credit for assumed validation efforts made by the application range, may be considered suitably validated. However, software manufacturer of purchased software configuration/ modifications should be but requires that individual validated as in 5.4.7.2 a)” spreadsheets, macros, and all configuration / modifications / setups be validated. This does not apply to electronic systems used to acquire, manipulate or reduce data, such as hard-coded firmware  that is often supplied with computer-driven devices. “Equipment, and its software…….shall Validation and Maintenance of be capable of achieving the accuracy Electronic system required……. Before being placed in Does the accuracy of the service, equipment (software) shall be Firmware/Software meet or exceed calibrated or checked to establish that it the accuracy required by the test meets the labs requirements·…….” method or other relevant specification? A good test is the uncertainty contribution of the device (and its programming) to the overall uncertainty of the test result. All deployed software should be verified prior to being placed in service by performing some user acceptance testing such as a comparison of the requirements against features. This includes placement into service following a move or after being shipped back from calibration or maintenance. “Each item of equipment and its Maintenance of Electronic system software used for testing… …shall… Each item of equipment & software …be uniquely identified.” shall be uniquely identified. Procedures should include documenting the versions in use (version control) “Records shall be maintained…” Maintenance of Electronic system Records shall be maintained of equipment & software.


Issue No. 001

Page: 11/17

Clause Extract / Wording 5.5.11 “Where calibrations give rise to… …correction factors… …procedures to ensure that copies (e.g. in computer software) are correctly updated.”

5.5.12

5.10.1

5.10.2.j

5.10.7

Policy Consideration Validation of Electronic system Does evidence exist confirming correct software deployment at each target installation? Consider the same approach to software as for other documents such as document control (software management), distribution control. Test and Calibration equipment, Integrity of data and Maintenance including software, shall be safeguarded of Electronic system from adjustments…” Software shall be safeguarded from adjustments such as password protection on spreadsheets or other files. NOTE 2 Integrity of data “The test reports or calibration Reports may be issued electronically certificates may be… ….by electronic data transfer…” “the… …identification of person(s) Integrity of data authorizing the test report or calibration Reports may contain electronic certificate.” signatures. LIMS systems should have established authorization protocols. “in the case of transmission of test or Integrity of data calibration results by… …electronic… Reports may be transmitted …means, the requirements of this electronically. Whatever method of International Standard shall be met…” transmission is used, it must provide an the same level of protection of integrity of information afforded to paper documentation.


Issue No. 001

Page: 12/17

APPENDIX 2 VALIDATION OF MS EXCEL AND OPENOFFICE.ORG CALC SPREADSHEETS

MS Excel is one of the most commonly used computer programs to perform automatic or semiautomatic calculation and visualisation of data. Excel spreadsheets may be used whenever a standard calculation has to be repeated over a long period of time. It should be noted that spreadsheets are programs, and that they as such require validation. The validation of spreadsheets should ensure that all data generated and all operations and checks performed with the spreadsheet, as well as online-generated graphs, are valid. Excel offers several useful tools for spreadsheet validation, e.g.: - data validation:


Issue No. 001

Page: 13/17

-conditional formatting:

or formula auditing:


Issue No. 001

Page: 14/17

which displays relationships between cells.

OpenOffice.org Calc offers Similar tools, though the functions may have different names: - data validation:


Issue No. 001

Page: 15/17

- conditional formatting:


Issue No. 001

Page: 16/17

and formula auditing.

More details about validation of Excel spreadsheets can be found in: Analytical Method Validation and Instrument Performance Verification Chung Chow Chan (Editor), Herman Lam (Editor), Y. C. Lee (Editor), Xue-Ming Zhang (Editor) ISBN: 0-471-25953-5. (http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471259535.html).


Issue No. 001

Page: 17/17

QCC ITSV 001 001 IT Software Validation[1]

Recommend Documents