Emergency Shutdown System Testing

Engineering Encyclopedia Saudi Aramco DeskTop Standards

EMERGENCY SHUTDOWN SYSTEM TESTING

Note: The source of the technical material in this volume is the Professional Engineering Development Program (PEDP) of Engineering Services. Warning: The material contained in this document was developed for Saudi Aramco and is intended for the exclusive use of Saudi Aramco’s employees. Any material contained in this document which is not already in the public domain or may not be copied, disclosed to third parties, otherwise used inreproduced, whole, or insold, part, given, withoutorthe written permission of the Vice President, Engineering Services, Saudi Aramco.

Chapter : Process Instrumentation File Reference: PCI-106.05

For additional information on this subject, contact PEDD Coordinator on 874-6556

Engineering Encyclopedia

ESD Systems Emergency Shutdown System Testing

CONTENT

PAGE

INTRODUCTION............................................................................................................3 REQUIREMENTS FOR EMERGENCY SHUTDOWN SYSTEM (ESD) TESTING.........4 Types of Tests .....................................................................................................5 Relationship of Tests to Project Execution ..........................................................5 REQUIREMENTS FOR FACTORY ACCEPTANCE TESTING (34-SAMSS-623)..........6 Purpose for Factory Acceptance Tests................................................................6 Design Document Requirements for Factory Acceptance Tests .........................7 Test Equipment Requirements for Factory Acceptance Tests.............................9 Procedure for Conducting Factory Acceptance Tests .......................................10 Software Error Detection — Duane Plots................................................15 REQUIREMENTS FOR SITE ACCEPTANCE TESTING .............................................16 Purpose/Requirements for Site Acceptance Tests ............................................16 Design Document Requirements for Site Acceptance Tests .............................17 Test Equipment Requirements for Site Acceptance Tests ................................18 Procedure for Conducting Site Acceptance Tests .............................................19 REQUIREMENTS FOR PROOF TESTING (34-SAMSS-623) .....................................21 Purpose/Requirements for Proof Testing ..........................................................21 Specific Proof Testing Requirements ................................................................22 Self-Diagnostics......................................................................................23 Frequency of ESD Testing......................................................................24 Procedures for Proof Testing.............................................................................27 SOE Testing and Resolution Requirements ...........................................31 Bypassing of ESD Inputs and Outputs....................................................31

Saudi Aramco DeskTop Standards

i



Typical Applications ................................................................................32 Authorization Procedures........................................................................32 Documentation........................................................................................33 Logging Procedures................................................................................33 GLOSSARY .................................................................................................................34 ADDENDUM: DUANE PLOTS .....................................................................................39

LIST OF FIGURES

Figure 1. Typical Test Equipment for a FAT...................................................................9 Figure 2. PLC-Based ESD System Interfaced To BPCS.............................................. 13 Figure 3. Types 1, 2, and 3 Error Descriptions.............................................................40 Figure 4. Duane Plot for ESD System 1 .......................................................................44 Figure 5. Duane Plot for ESD System 2 .......................................................................45


ii



INTRODUCTION

This module is a natural progression of the previous modules. Module 1 discussed emergency shutdown (ESD) systems and their role in an operating plant, some Saudi Aramco mandatory requirements that govern the design and use of ESD systems, the basic structure of an ESD system, and typical technologies that are used in ESD system. Module 2 discussed documentation requirements for an ESD system. Module 3 discussed design requirements and application criteria for an ESD system that can be used to determine if an ESD system meets Saudi Aramco requirements. Module 4 discussed the necessary background that is needed to make changes to an ESD system that is installed and operating. Changes to the following three areas of an ESD system were discussed in Module 3: •

Input devices

•

Logic solver and associated application programs

•

Output devices

This module discusses necessary testing of an ESD system to establish and maintain the integrity of the ESD system. All three areas of the ESD system that were discussed in Module 4 must be tested.


3



REQUIREMENTS FOR EMERGENCY SHUTDOWN SYSTEM (ESD) TESTING

The integrity of an ESD system operation must be ensured prior to the start-up of the plant, during normal operation, and after any modifications have been made. In PES-based equipment, software programs should be validated against the design specification to ensure plant safety. Validating that an ESD system performs the required functions in a safe manner can best be accomplished through testing. There are at least three areas that testing of ESD systems should cover: •

Testing of the ESD system hardware (including all interconnections of the components).

•

Testing of the written software (application program).

•

Testing of the process operation performed by the system (functional test).

The hardware test should verify the correct physical and soft (communications link) connections of all inputs and outputs associated with the ESD system. This test includes the primary sensors, I/O interface devices, the logic solver, and the final shutdown devices. Testing of the written software (application program) should include a review of the program logic by someone not directly associated with the program development. Simulation of the program using either the actual system or other PES-based equipment is required by Saudi Aramco in order to ensure an error-free program. During project execution phases, ESD application program functional tests must be witnessed and validated by representatives from respective proponent engineering, maintenance, and operating departments. During project execution phases, project records must be kept that document all ESD logic, input device and final element testing, as well as test results and all problem resolutions.


4



Where practical, ESD systems should be designed to permit functional testing of field device inputs, internal logic, and final shutdown devices without requiring the complete bypassing of the ESD system or a process unit shutdown in order to accomplish the test. Types of Tests

The following three types of testing are normally used for an ESD system: •

Factory acceptance test (FAT)

•

Site test

•

Proof test

The factory acceptance test is used to verify the integrity of the ESD system prior to shipment from the ESD system vendor. The site acceptance test is used to validate the operation of the complete ESD system after installation at the plant site. The proof test is used to maintain the integrity of the ESD system during operation and maintenance of the ESD system. Relationship of Tests to Project Execution

Factory acceptance testing is normally performed while the process system is being constructed. The sensing devices and final devices are installed in the field during the construction period of the project. Because the FAT is performed at the ESD system vendor location, the ESD field devices (i.e., sensors and final devices) are not tested during the FAT. Site acceptance testing is performed after the construction phase of the project is complete. This phase of the project is often called the commissioning phase. During the commissioning phase, all parts of the process system, including the ESD system, are tested. Site acceptance testing continues into the start-up phase of the project. Proof testing takes over once the start-up phase of the project has been completed. This phase coincides with the operating portion of the process system.


5



REQUIREMENTS FOR FACTORY ACCEPTANCE TESTING (34-SAMSS-623)

Factory acceptance testing is conducted at the ESD system vendor's site so that any problems with the integrity of the ESD system can be detected and corrected before the ESD system is shipped to the plant site. The following are some of the advantages of performing these tests at the vendor's site: •

Vendor personnel who have been involved with the design and construction of the ESD system are readily available in case any problems are encountered.

•

Necessary modifications to the ESD system can be made more easily and with less expense if they are done at the vendor's location.

•

Commissioning and start-up of the process system will proceed more smoothly if the ESD system logic solver has all of the bugs worked out of it prior to delivery to the plant site.

Purpose for Factory Acceptance Tests

The purpose of the FAT is to test both the software and hardware functionality of an ESD system as an integrated unit. These tests involve the following: •

Hardware qualification and testing.

•

Software qualification, testing, and documentation.

•

Complete I/O and internal system wiring checkout, including tag number identification validation.

The goal is to identify and resolve any problems in the system before it arrives at the plant site. Saudi Aramco’s primary concern is to identify any potential common cause/mode faults that might compromise the integrity of the ESD system. These potential common cause/mode faults include: •

Improper ESD system grounding.

•

Field signal wire shielding and grounding locations.


6



•

Field power supply sizing, protection, distribution, and fuse coordination (if applicable).

In addition, Saudi Aramco tests and monitors the effect of voltage and/or current transients and possible RFI interference (from handheld transceivers) on I/O signals and CPU instruction/command processing. When such faults are identified, all possible failure modes are identified and evaluated, including the resulting ESD system action. There may be some areas where some specific problems cannot be corrected until receipt of the ESD system at the plant, but these cases should be kept to a minimum. During the FAT test, the complete ESD system, including all composite modules and interconnecting wiring, must be subject to both hardware and software functional tests. These tests must demonstrate the functionality of each individual component module within the integrated ESD system, including individual I/O point tests. The FAT should include testing of all hardware components and software in the system. Saudi Aramco performs complete loop testing through to the DCS and to the operator’s console. The FAT may be accomplished by either the vendor performing the testing, the user performing the testing, or a combination of the two performing the testing. The latter approach is the most desirable. The FAT will ensure that no surprises will be found upon installation, and that the system will perform the specified functions. Design Document Requirements for Factory Acceptance Tests

A complete set of design documents is needed in order to conduct a factory acceptance test. A cause-and-effect matrix, a written description of the ESD system functionality, and annotated logic diagrams (binary logic diagrams and/or ladder logic diagrams) should be used as the basis for a factory acceptance test of all vendor-supplied ESD system equipment.


7



The purchase order for the ESD system should list the additional design documents that are required for the FAT. The following are typical documents that are needed to effectively accomplish a FAT: •

The ESD system design specification that was used as the basis for the vendor's proposal.

•

A system arrangement drawing(s) for the ESD system that identifies each module type, location, and tag name.

•

An ESD system I/O list that shows all input and output devices.

•

Termination lists and/or wiring drawings that show where all external wiring is terminated.

•

Graphic design drawings (where appropriate).

•

Vendor manuals.

For PLC-based ESD systems, the following design documents are also needed: •

An annotated printout of all programs or program files in ladder logic format. To facilitate ESD system troubleshooting, the ladder logic printouts must include completed I/O addresses and logic element parameter identification.

•

An index of the system's data base including tag name(s), descriptors, and initial values.

•

I/O and internal element cross reference tables.

•

An event log configuration file/record (if so specified).

The FAT should not only check the functionality of the system, but should also check the accuracy of the documentation. At the end of the FAT, all documentation should accurately describe the system. Any exceptions should be included on a punch list.


8



Test Equipment Requirements for Factory Acceptance Tests

A wide variety of test equipment may be needed to effectively test an ESD system. The test equipment list should include all items required to perform 100% of the test. The list should also include equipment required for troubleshooting. The following is an example of the typical test equipment that is needed for the FAT is shown in Figure 1.

Device

Purpose

Digital multi-meter

Monitor system outputs and troubleshoot wiring

DC mA source

Input current signals (instrument simulation)

DC mV source

Thermocouple or other low-voltage input simulation

Pulse generator

Input pulse signals

Pulse counter

Monitor pulse signals

Resistance box

RTD (resistance-temperature detector) simulation

Variable DC voltage source

Input voltage signals (instrument simulation)

Breakout box

Troubleshoot data communication links

Simulation panel with capability of inputting discrete signals and indicating discrete output values

I/O simulation

Figure 1. Typical Test Equipment for a FAT


9



Procedure for Conducting Factory Acceptance Tests

A FAT testing procedure should be developed that outlines the details to be addressed in the test. As a minimum this procedure should include the following: •

State the location and dates of the FAT.

• •

Provide a description of the general approach. Provide a description of the format of the FAT punch list.

•

Specify the revision levels of the hardware and software to be tested.

•

Specify the exact configuration of equipment being tested.

•

Address personnel safety issues that may apply during the test.

Documentation of the FAT testing procedure and results is important in order for the FAT to accomplish its intended purpose. The detailed test procedure should ensure that all aspects of the system are checked against system documentation. The test procedure should at least include the following: •

A description of a typical loop test for each type of I/O in the system using the proper test equipment. Inputs should be simulated at 0%, 25%, 50%, 75%, and 100% signal input. Outputs should be monitored at 0%, 25%, 50%, 75%, and 100% of the output level.

•

A description of a typical method for testing the ESD system logic.

•

A description of what checks are to be made on graphic displays.

•

Provision for a method for checking all other aspects of the system (i.e., visual checks, trends, logs, system failures).


10



Simulation can be a useful technique for verifying the operation of an ESD system. Some of the objectives for using simulation are: •

•

Verify that the program functions as shown in the logic diagrams. Review and verify that the program accomplishes all of the process objectives.

•

Evaluate the program from a safety standpoint.

•

Train both operating and maintenance personnel.

The simulation should allow viewing of all PLC controlled devices simultaneously. A "switches and lights" type of simulation is generally not acceptable. A "true" simulation is desired where: •

The application program outputs manipulate a simulator's inputs.

•

The simulator program duplicates the action of the actual process as closely as possible.

•

The simulator program outputs an input signal to the application program.

A typical PLC-based ESD system is shown in Figure 2. This ESD system interfaces to a BPCS using digital communications. The following is a typical procedure for performing a factory acceptance test on the ESD system: •

Using the system arrangement drawing(s) and the design specification, verify that the correct components are installed.

•

Using the termination list and/or wiring drawings, verify that all wiring has been terminated and identified correctly.

•

Perform a tug test of all wire terminations by physically stressing eachand wireterminated termination to determine whether it has been crimped properly. The intent is not to break wiring or stress insulation but to test the integrity of the termination.


11



•

Connect the simulation panel to the ESD system.

•

Turn power on to all equipment in the ESD system.

•

Activate all inputs using the simulation panel and/or other test equipment (e.g., DC mA source). Using the programming device, verify that the input module actuates and that the correct address is actuated in the ESD system. Test all the various types of input devices that are shown in Figure 2, including analog, digital, and discrete inputs.

•

Using the programming device, activate all outputs. Verify that the correct output module actuates and that the appropriate final device is turned on using the simulation panel, other test equipment, or other component of the ESD system (e.g., annunciator). Test all types of output devices as shown in Figure 2.


12



Operator Consoles

Connection To Network

Local Area Network Connection To Network

Dedicated Critical Alarm Annunciator

Alarm Horn

BPCS Logic Solver

MODBUS

ESD System Logic Solver PLC-Based With TMR Technology (See Note 1)

ESD Gateway

MODBUS

Input/Output Wiring (Opto-Isolators Used For Signal Replication)

Sequence-of-Events Recorder (Event Logger)

AS

S

FO M

XSL

M

Discrete Output Devices (e.g., motors)

Discrete Input Devices (pushbuttons and other switches)

XSH

Automatic Block Valves

Analog Sensing Devices (transmitters, thermocouples, and RTDs)

Figure 2. PLC-Based ESD System Interfaced To BPCS


13



•

•

Load the application program into the ESD system. Using the cause-and-effect matrix, a written description of the ESD system functionality, and annotated logic diagrams (binary logic diagrams and/or ladder logic diagrams) as the basis, verify the correct operation of all interlock circuits. Actuate input modules using the simulation panel and/or other test equipment. Verify the correct operation of final devices using the simulation panel, other test equipment, or other component of the ESD system. Test all other external communications where possible, such as the MODBUS digital communications device shown in Figure 2. Where possible, functionally test the communications interface using actual cable types and intended cable lengths.

•

Verify that all vendor-supplied diagnostic routines (e.g., internal watchdog timers) function by simulating CPU failure, I/O module/individual point failures; power supply failure, communications interface failures, and card replacement-induced failures. One method for accomplishing this testing is by fault injection testing (i.e., creating failures by disconnecting components, shorting inputs or outputs, and/or cutting power to components).

•

Verify all event logging functions by randomly generated input event cycling, with the specified point resolution being demonstrated.

•

If a true simulator is being used, connect the simulator to the ESD system. Repeat the tests on the application program and verify that the ESD system responds as expected. Some modifications to the ESD system application program may be necessary, such as changing the timer settings on valve travel alarms.

•

Test fault histories/summaries by logging and annunciating both on an external printer and an operators console.

•

Develop a punch list that documents all items that do not adhere to the design specification.

•

Document all failures in the application program. These failures may be used to develop a Duane Plot to ensure software reliability (see next section and Addendum).


14



All discrepancies noted in the punch list must be resolved to the satisfaction of Saudi Aramco. Results of the FAT test must be documented by a written report, supported by the FAT procedure used. When the FAT has been completed, all design documentation must be updated. The equipment may be released for shipping at the conclusion of the FAT. Prior to shipment of the ESD system, all ESD modules must be removed from chassis and located in separate boxes/containers. An official acceptance document should be signed by both the user team leader and by the vendor. The acceptance document should state whether open items on the punch list are to be resolved in the field or prior to shipping. The user should have the right to return to the factory to back-check any items agreed to be resolved at the factory. Software Error Detection — Duane Plots

Although not required during FAT testing, a Duane Plot can sometimes (see be used to showFrom the status of softwarethat error detection Addendum). the information is collected as part of the Duane Plot methodology, the following can be determined: •

If progress is being made towards a stated reliability factor for the system.

•

A prediction of the testing time required until the next software error is found.

•

A prediction of the number of errors that will be found in a stated period of testing time.

•

A prediction of how many more hours of testing will be required to reach the desired reliability.

Meticulous record keeping is required in order to capture all software errors.


15



REQUIREMENTS FOR SITE ACCEPTANCE TESTING

The integrity of the complete ESD system must be validated after the ESD system has been installed at the plant site. This site acceptance test is the first test that is conducted on the ESD system with all field devices connected. Purpose/Requirements for Site Acceptance Tests

The purpose of the site acceptance test is to achieve the following: •

Verification of all inputs to a system for proper termination assignments, functionality, ranges, etc.

•

Verification of application programming of the system through functional or simulation tests.

•

Verification of proper operation of all outputs from the system.

•

Diagnostic tests on system hardware and software.

•

Verification of the accuracy of all custom graphic displays with associated data.

•

Verification of controller cycle time.

Specific requirements for the site acceptance test are to verify the following: •

The operation and range of all input devices including primary sensors and shutdown system input modules.

•

The logic operation associated with each input device.

•

The logic associated with combined inputs where appropriate.

•

The trip initiating values (set points) of all inputs or the

•

contact position of all switch inputs. The alarm functions that may be included.

•

The operating sequence of the logic program.


16



•

The function of all outputs to final control elements.

•

The correct action of the final control elements.

•

The first out alarms, if appropriate.

•

Any variable or output status indications that might be provided for operator monitoring (e.g., printed messages).

•

Any computational functions performed by the shutdown system.

•

The emergency switch provided external to the shutdown system logic program works to bring the system to its "failsafe" condition.

•

System action on loss of electrical power, both instrument and utility power.

•

The "fail-safe" status of all inputs, outputs, and final control elements (e.g., thermocouple burnout and valve failure position).

Design Document Requirements forTests Site Acceptance

The design document requirements for site acceptance tests are very similar to the design document requirements for the factory acceptance test. The following are typical documents that are needed to effectively accomplish a site acceptance test: •

The ESD system design specification that was used as the basis for the vendor's proposal.

•

A system arrangement drawing(s) for the ESD system that identifies each module type, location, and tag name.

•

An ESD system I/O list that shows all input and output devices.

•

Termination lists and/or wiring drawings that show where all external wiring is terminated.

•

Graphic design drawings (where appropriate).


17



•

Vendor manuals.

•

Field wiring diagrams.

•

Specification sheets for field instruments.

•

Installation detail drawings for field instruments.

For PLC-based are also needed:ESD systems, the following design documents •

An annotated printout of all programs or program files in ladder logic format. To facilitate ESD system troubleshooting, the ladder logic printouts must include completed I/O addresses and logic element parameter identification.

•

An index of the system's data base including tag name(s), descriptors, and initial values.

•

I/O and internal element cross reference tables.

•

An event log configuration file/record (if so specified).

At the end of the site acceptance test, all documentation should be updated so that it accurately describes the system. Test Equipment Requirements for Site Acceptance Tests

The test equipment requirements for site acceptance testing are somewhat different that the test equipment requirements for factory acceptance testing. Because all field devices are connected to the ESD system, simulation panels are not needed. Some of the other test equipment for simulating inputs into the ESD system are also not required. However, some additional test equipment is needed to simulate input signals into transmitters and switches. Sensing devices should be actuated by simulating process conditions at the sensing element where possible. Where the actual process cannot be used for the test, as in the initial phase of the site acceptance test when no process


18



materials are in the system, simulated transmitter and switch signals are used. Some examples are shown below: •

•

Thermocouples would be simulated by a millivolt generator, and RTDs would be simulated by a resistance box. Pressure transmitters would have pressure loaded into their process connections using a pressure calibrator.

•

Capacitance or conductivity level probes would be immersed in a bucket of liquid with a similar dielectric constant and conductivity instead of actuated by turning the sensitivity dial.

Test equipment must be available to provide input signals into sensing devices that represent a process condition as accurately as possible. Procedure for Conducting Site Acceptance Tests

The first phase of site acceptance testing is off-line testing, which means that the process is not operating. Because the process is not operating, a complete and detailed test can be conducted. Off-line testing should be performed on all new systems prior to placing them in operation. The following is a typical procedure for performing a site acceptance test on the ESD system: •

Using the design specification, specification sheets, and installation detail drawings, verify that the correct field devices are installed and that they are installed properly.

•

Using the termination list and/or wiring drawings, verify that all field devices have been terminated at the correct locations in the logic solver and that the wiring from these field devices has been identified correctly.

•

Check all wiring, particularly field wiring, to verify that there are no shorts to ground.

•

Turn power on to all equipment in the ESD system.


19



•

•

Activate all inputs using the test equipment provided. Using the programming device, verify that the input module actuates and that the correct address is actuated in the ESD system. Test all the various types of input devices that are shown in Figure 2, including analog, digital, and discrete inputs. Using the programming device, activate all outputs. Verify that the correct final device actuates and that the failure position of the final device is correct.

•

Load the application program into the ESD system. Using the cause-and-effect matrix, a written description of the ESD system functionality, and annotated logic diagrams (binary logic diagrams and/or ladder logic diagrams) as the basis, verify the correct operation of all interlock circuits. Actuate inputs using the test equipment provided. Verify the correct operation of final devices.

•

Test all other external communications where possible, such as digital communications device to a BPCS.

•

Whenever possible, the ESD system should be exercised prior to start-up with a "dry run." With a dry run, process materials do not constitute a safety These hazardprocess (e.g., water or oil) are that pumped through the system. materials can be used to simulate actual process conditions at the inputs of sensing devices.


20



REQUIREMENTS FOR PROOF TESTING (34-SAMSS-623)

Testing the correct functionality that is performed by the ESD system is generally referred to as a proof (or functional) test. This proof test provides validation that the ESD system logic controls the action of the final devices as specified by the design specifications. Purpose/Requirements for Proof Testing

The purpose of proof testing is to ensure the integrity of the ESD system. The proof test must be designed to verify each function of the interlock logic and the interactions of the various components to uncover any problem areas that might exist. The following are some requirements for proof testing: •

The functional testing should be done in a formal manner by a team of technical, operations, and maintenance personnel who have a working knowledge of the system being tested.

•

A written procedure should be used that describes each

•

step that is to be performed during the test. The written test procedure should be specific to each interlock in the ESD system.

•

The written procedures must include instructions for any bypassing or jumpering necessary for testing, and they must provide assurance for removal of such bypasses and jumpers. Bypassing safety devices should be avoided whenever possible, and bypassing should only be done with proper management notification or compliance with the applicable permit system.

•

Formal documents should be used for recording the results of the proof testing.

•

Only those persons who have proper knowledge of the system and the process should be allowed to perform any on-line tests on operating plants.


21



•

The necessary test equipment for performing the functional checkout of a shutdown system must be determined prior to the time for the checkout, and this test equipment must be defined in the written procedure. Any special equipment requirements must be identified. The proof test checkout procedure should list the required equipment, by manufacturer and model number where appropriate, and the number of test equipment items needed to perform the test. Provision for maintenance and storage of test equipment used for proof testing should also be made.

•

Provision must be made for verification of test equipment against traceable standards to ensure accuracy of the test equipment. This method should be documented for reference.

•

A mitigation plan should be prepared for each interlock that defines the actions that should be taken whenever any part of the interlock is found to be inoperative or incorrect.

Specific Proof Testing Requirements

Each plant should have a written program that identifies critical emergency shutdown devices that exist in the plant, the frequency of testing required, the method of testing, the responsibility for testing, and the responsibility for administration of the testing program. The program should include some method of automatically notifying the person responsible for conducting the test. In general, each plant location should have a system in place that provides the following: •

A list of all interlocks by classification and the equipment that is included in each interlock.

•

A concisely written description of the purpose and function

•

of each interlock that is included in the testing program. A planned test interval for each system.


22



•

A call-up system to schedule testing and track compliance to this schedule.

•

A system to track and monitor test results along with identification of specific problems found.

•

A system that looks for continuing problems with systems under test. A mechanism must exist for investigating these problems and to ensure that appropriate correction action is taken to eliminate the problem. Possible solutions may include hardware modification, additional preventive maintenance, or adjustment to test intervals.

Field testing of devices in normal service should be employed where practical. All test methods should, where practical, simulate actual operating and/or upset conditions. Self-Diagnostics

Vendors of PLC-based ESD systems build self-diagnostics into their systems. One common method that is used is an internal watchdog timer that monitors the operation of the system to ensure that inputs are being scanned, that the application program is being executed, and that outputs are being written to the output devices. Some manufacturers also build self-diagnostics into their input modules and output modules. An example of self-diagnostics for an output module is to have the output module pulse the output on or off for a short period of time to ensure that the output in fact does turn on or off. Built-in diagnostics in some output modules can detect a triac failure and switch to a backup triac. PLC-based TMR (triple modular redundant) systems use comprehensive fault detection methods using 2 oo3 voting and fault detection circuits in both firmware and software. These circuits automatically identify, alarm, isolate, and contain faults without compromising ESD system performance.


23



Frequency of ESD Testing

The frequency of testing required for ESD systems is dependent on the safety protection that the ESD system provides. If the safety function that is performed by the safety interlock is not critical, the testing interval can allow longer periods between tests. If the safety function that is performed by the safety interlock is critical, the testing may have to be done more frequently. Another factor that can impact the testing frequency is the finding of faults or failures of any system components during a test. The number of failures may dictate more frequent testing, or the lack of failures could allow longer intervals between tests. There should be a balance, however, between the time taken for testing and the estimated time the equipment will be out of service due to failures. In no instance should the frequency of testing be less than that recommended by the HAZOP team. Testing is important because testing can increase the availability of the ESD system. A useful definition of system availability is: A = Uptime/Total Time where A = availability. Availability is measured by the probability that the system is working throughout the total mission time. If it is always working, the availability is 1.0. Multiplying the availability by 100 percent permits expression of availability as a percentage; a perfect system has 100 percent availability. An ESD system with 100% availability would always respond when a demand is imposed on the ESD system, and the ESD system would take the necessary corrective action to take the process and/or equipment to a safe state. Before discussing availability further, it is important to look at the types of faults that are experienced in ESD systems. The types of faults experienced can be divided into fail to safe (FTS) and fail to danger (FTD). Fail to safe faults will result in an immediate system shutdown; they signal their presence. These types of faults are called


24



"revealed" faults or "overt" faults. These types of shutdowns can be dramatically reduced by using redundant elements within the system, because control is maintained if one element becomes faulty. Fail to danger faults are the most dangerous. Fail to danger faults prevent the system from responding to hazard warnings, allowing hazards to develop. These faults are called "unrevealed" faults or "covert" faults, because they can remain undetected until revealed by testing. If the fault remains "unrevealed" due to lack of testing, the system will not be available when a demand arises. With testing, very high degrees of protection can be achieved, and the shorter the time interval between tests, the smaller will be the probability of two faults existing in different elements of the system. Availability can also be defined in terms of the mean time between failures (MTBF) and the mean downtime (MDT): A = MTBF/(MTBF + MDT) MDT is really a summation of the mean time to diagnose the presence of a system fault (MTDF) and the mean time to repair (MTTR): MDT = MTDF + MTTR MTTR can be broken down into: MTTR = MTDL + MTRF + MTRO where: MTDL = mean time to determine a fault location MTRF = mean time to replace a faulted component MTRO = mean time to return the system to operable condition


25



The MDT values can be developed from considerations of the following: •

The location of repair personnel

•

The average repairman's skill level

•

The ease of diagnosis of the system fault

•

The accessibility of spares.

MDT values can be combined with the vendor's quoted MTBF numerics to produce availability values for an ESD system or a particular component of an ESD system. For FTS faults, MTDF = 0, because the fault is self revealing. A = MTBF/(MTBF + MTTR) For FTD faults, MTDF is most important, and it often determines the overall availability, because this term is usually much larger than the MTTR. Therefore: A = MTBF/(MTBF + MTDF + MTTR) The MTDF is a function of how often the system is tested, or the test interval (TI). The test interval is the time interval between two successive tests. A FTD fault can occur any time during the test interval. On the average, the failure can be assumed to occur about the middle of the test interval or1/2 TI. Therefore, A = MTBF/(MTBF + 1/2TI + MTTR) If the system is tested manually, the test interval tends to be much longer than MTTR, and A = MTBF/(MTBF + 1/2TI) = MTBF/(MTBF + MTDF) Because the test interval is in the denominator of this equation, decreasing the test interval (i.e., increasing the frequency of testing) increases the availability of the ESD system.


26



Procedures for Proof Testing

Proof testing should be performed prior to initial operation of an ESD system for all new installations. Proof testing should be repeated for all modifications prior to their initial operation. Proof testing should also be repeated, in total, after all major turnarounds where work has been done that might impact any ESD system components. The proof testing after minor maintenance or minor modifications to an ESD system may not require the same level of testing that would be required for initial validation or after major modifications. Procedures should establish whether or not the ESD system is still capable of meeting the design specifications by appropriate testing. Some sound engineering judgment will obviously be required in this area. Proof testing may be performed off-line or on-line. Off-line proof testing is basically the same as a site acceptance test. The process is not operating, so some input signals must be simulated. On-line testing is much more difficult because it is performed while the process is operating. This type of testing requires special safety considerations because any unexplained or inappropriate actions that the test might precipitate could result in a potentially hazardous event. Plans should be developed and approved, prior to any testing, that describe the following: •

The purpose of the test

•

The test procedure

•

The persons performing the test

•

The expected results of the tests

•

Any special precautions that may be required during the test to ensure safe operation of the plant.


27



This plan should include one of the following two options: •

Immediately shut down the portion of the unit that is protected by the interlock.

•

Immediately implement a predefined plan that insures protection is provided by other means during the time interval the interlock is out of service, and the plant continues to operate. The development of this plan may need the assistance of plant personnel that are familiar with plant operating and maintenance practices.

Another concern is related to testing ESD systems where portions of a single channel system (entire ESD system logic included in a single processor) require testing while the processing unit continues to operate. This test would require that the ESD system either be out of service or partially bypassed during the testing. Special precautions should be taken if testing of this nature is attempted to ensure that adequate protection is constantly in place for the unit. Concerns that may require special attention include: •

•

How testing of a portion of the system can be accomplished safely and without potential for inadvertent changes to remainder of system. Means of bypassing only the logic being tested.

•

Existence of monitoring of key variables being tested by some other techniques, direct or inferred, during the testing.

•

Operating conditions that might need to be adjusted for the testing to take place safely.

Each ESD system is an independent system that will require its own testing procedure. There may be some synergy between parts of other systems, but each ESD system should have its own, written and approved, proof test procedure.


28



Typical items that may be included in the detailed test procedures are: •

Interlock name and class.

•

Initial and present test frequency.

•

Purpose and action of the interlock (includes a description of the hazard, trip point process and signal and denotes energization or in de-energization aboveunits or below set point).

•

Instrument and electrical drawing and specification numbers with latest revision numbers for reference.

•

A simplified P&ID.

•

A simplified functional block diagram of any software calculations.

•

Descriptions of test methods for sensors, transmitters, switches, software, and final devices.

•

Dates and signatures of maintenance and operations supervisors who approved the procedure.

•

Dates and signatures of people who conducted tests.

•

The pass or fail status of such tests.

•

Exception reports for any test failures.

Some care must be taken when doing proof testing. Deliberately imposing a demand on a system is obviously undesirable, because if the system fails, the demand could cause the incident the system is designed to prevent. For example, steam boiler low level trip tests are often conducted by deliberately lowering the drum level. This questionable practice has resulted in more than one ruined boiler when the trip failed, and the attendant for whatever reason failed to respond.


29



If the demand is not imposed, how do you ensure that the sensor is capable of detecting it? If the sensor is isolated from the process and an artificial demand imposed, how do we guarantee it is not left isolated or that the impulse lines are not blocked? A high temperature interlock could be tested by removing it from the thermowell and immersing it into a container of liquid at appropriate temperatures. However, this test does not check that the thermowell is clean and free from process side buildups that could seriously impair the system response time and, consequently, the effectiveness of the trip. The other disadvantage of this type of test is that damage might be caused to the temperature element when reinstalling it into its correct position. In systems where there is redundancy, simply verifying that the final result is obtained is totally inadequate. For example, duplicate shutoff valves may have been installed for added reliability. A test that verifies that the flow stops, therefore, is not adequate because only one valve may have closed, and the other one could be in a failed dangerous condition. The test should verify correct operation of all components. Testing an interlock has little value if the demands are more frequent than the tests. Testing must increase the availability of the ESD system. The amount of time that the ESD system is not available is that time when the system is incapable of providing the protection for which it was designed. The time that the ESD system is not available consists of: •

The time when the ESD system is in a failed danger state. The way to minimize this time is to use reliable equipment in a well designed and installed system and to test frequently so that the fault is found soon after it occurs.

•

The time when the ESD system is bypassed or isolated for the test. The way to minimize this time is to test infrequently, which compromises the first bullet item, or to do the test quickly which may compromise its thoroughness and effectiveness.


30



•

The time when the ESD system is left bypassed or isolated after the test. Minimizing this time is not affected by the frequency of testing. Each test represents an opportunity for a mistake, and if the system is left isolated, this mistake probably will not be discovered until the next test. So frequent testing means more opportunities for the mistake but a mistake of a shorter duration, but less frequent testing means less opportunities for the mistake but a mistake of longer duration. The way to minimize this problem is to reduce the probability that the tester will make the mistake.

SOE Testing and Resolution Requirements

The frequency required to test a sequence-of-events (SOE) recorder can be difficult to achieve during proof testing. One method that can be used is to simulate the input to the SOE recorder using a pulse generator. A wide array of frequencies is possible with these devices. Even SOE recorders that have a requirement for 100 millisecond resolution can be tested with the proper pulse generator. Bypassing of ESD Inputs and Outputs

ESD systems must include provision for the proof testing requirements. If on-line testing is to be required, test points or other means should be provided to eliminate the need for removing and replacing wires during the testing. The need for any bypasses required for testing should also be addressed during the design phase with the ultimate goal of eliminating bypasses wherever possible. One means for providing the test points would be to include additional terminal connections on the inputs and outputs of the ESD system equipment with capability for attaching test equipment for testing.


31



Typical Applications

The following are some reasons for the installation of bypassing capability of an ESD system input signal: •

Startup permissives on initiating variables.

•

On-line required calibration or maintenance work.

•

Approved interlock set point changes on initiating variables.

•

Preventing nuisance trips due to temporary signal noise or interference.

•

During proof testing of the ESD system while the process unit is not operating.

Output bypass switches for ESD system shutdown outputs must only be considered when no other mechanism is available for on-stream maintenance or testing of an ESD system without affecting associated process equipment. When a final device can be bypassed in the field or when a shutdown cabinet bypass can be used for testing ESD system (e.g., isolation valve movement) operation, an output bypass switch must not be used.

Authorization Procedures

All procedures that are related to the bypassing of ESD system functions should be approved in writing by appropriate plant management prior to use of the bypass. These procedures may include decisions that require thorough analysis, testing, documentation, and communication to appropriate personnel before they are implemented. A time limit that determines how long a bypass may be in place may be needed in some instances.


32



Documentation

When bypass switches, or some other bypassing method, are required, there should be a written procedure that prevents having more than one signal bypassed at the same time. All instances of bypassing should be documented, and the returnto-normal position should be a requirement, prior to signing off that any work has been completed. Where the ESD system has the capability, changes in positions of all bypass switches should be automatically logged by the system. This requirement is just as important for off-line as for on-line testing. Only those bypasses that are truly required for maintenance or testing should be allowed in the system.

Logging Procedures

Bypass procedures should also require special tagging on all ESD system input devices that are in a bypass mode during operation of the plant. The tags should be visible and should identify the following: •

The function bypassed.

•

When the bypass was initiated.

•

Who approved the bypass.

•

Personnel authorized to remove the tag.

The tags should not be removed until the system is returned to a normal operating mode. The time the tag is removed, and the individual removing the tag should be noted in the operations logbook.


33



GLOSSARY 2oo3 voting

A 2-out-of-3 redundant system that requires at least 2 of the 3 three channels to be in agreement before the ESD system can take action.

annotated logic

A graphical method for showing ESD inputs, outputs, and

diagram

internal logic using AND/OR, timer, or counter logic elements with basic logic statements embedded in the diagram.

annunciator

A hardware device or software application that is used to convey alarm information.

application program

Software that is specific to the user application in that it contains the logic program written to meet the overall requirements for the ESD system.

availability

The probability that a system will be able to perform its designated function when required for use. As used in this course, this term is an indication of an ESD system's ability to react when a demand is placed on the ESD system.

basic process control system

The control equipment and system that is installed to regulate normal production functions.

binary logic diagram

A method of representing the logic in binary interlock and sequencing systems using abstract logic functions such as AND, OR, and NOT.

BPCS

An abbreviation for basic process control system.

bypassing

Act of temporarily defeating a safety function in an ESD system.

cause-and-effect matrix A form of state table that is used for showing the relationship between a process input and an output device in binary interlock and sequencing systems. demand

A condition or event that requires a protection layer to take appropriate action to prevent and/or mitigate a hazard.

Duane plot

A methodology that can be used to show the status of software error detection.


34



ESD

An abbreviation for emergency shutdown system.

emergency shutdown system

A system composed of sensors, logic solvers, and isolation devices that takes the process to a safe state when predetermined conditions are violated.

factory acceptance test A test of an ESD system that takes place at the vendor's site and that does not test the field devices of the ESD system. fail-safe

A concept that defines the failure direction of a component or system as a result of specific malfunctions. The failure direction is toward a safer or less hazardous condition.

fail-to-danger fault

A hardware or software failure that inhibits or delays actions to achieve a safe operational state should a demand occur. This type of failure has a direct and detrimental effect on ESD system availability.

fail-to-safe fault

A hardware or software failure that causes the process and/or the equipment to go to a safe state. This type of failure has a direct and detrimental effect on ESD system reliability.

FAT

An abbreviation for factory acceptance test.

hazardous event

An occurrence related to equipment performance or human action, or an occurrence external to the system that causes system upset, that has the potential for causing harm to people, property, or the environment.

HAZOP

An abbreviation for hazard and operability study.

I/O

An abbreviation for input/output.

input bypass

A hardware or software method for defeating the action of an input device in order to test or maintain the input device.

input device

Discrete hard-wired, push or pull buttons; process (nonpowered) static switches; transmitter(s)/ actuated transducer(s) using a 4-20 mA DC current or digital transmission format; thermocouples; and RTDs that provide input signals to the logic solver in ESD systems.

integrity level

An indicator of ESD system performance.


35



ladder diagram

A diagram that uses symbols and a plan of connections to represent the logic in binary interlock and sequencing systems.

MDT

An abbreviation for the mean downtime.

mean downtime

The mean time that the ESD system is not able to respond to a demand once a fault occurs. The mean time between successive failures of a component or system.

mean time between failures

mean time to detect the The mean time that it takes to determine the specific location location of a fault of a fault. mean time to diagnose a fault mean time to repair

The mean time that it takes to determine that a fault has occurred. The mean time to repair a component of an ESD system. This mean time is measured from the time that a failure occurs to the time that the repair is completed and the ESD system has been returned to service.

mean time to repair a fault

The mean time that it takes to fix or replace a failed component.

mean time to return to operation

The mean time that it takes to return the ESD system to operable condition after a fault has been repaired.

mitigation plan

A plan that describes the actions that must be taken when a failed interlock is detected in order to reduce the consequences of the failure.

MODBUS

A digital communications technology.

MTBF

An abbreviation for mean time between failures.

MTDF

An abbreviation for mean time to diagnose the presence of a fault.

MTDL

An abbreviation of mean time to determine the location of a

MTRF

fault. An abbreviation for component.


mean time

to

replace a

failed

36



MTRO

An abbreviation for mean time to return to operable condition once the fault has been corrected.

MTTR

An abbreviation for mean time to repair.

on-line testing

Testing that is done while the process continues to operate.

output bypass

A hardware or software method for defeating the output of the the logic logic solver solver. in an ESD system in order to test or maintain

output device

Automatic block valves, motors, pilot lights, and similar devices that accept output signals from the logic solver in an ESD system.

P&ID

An abbreviation for piping and instrument diagram.

PLC

An abbreviation for programmable controller.

programmable controller (PLC)

A digitally operating electronic system, designed for use in an industrial environment, that uses a programmable memory for the internal storage of user-oriented instructions for implementing specific functions such as logic, sequencing, timing, counting, and arithmetic, to control, through digital or analog INPUTS and OUTPUTS, various types of machines or processes.

proof test

A test of all the components (i.e., hardware and software) of an ESD system to ensure that the system is capable of functioning when the demand arises.

punch list

Documentation that logs any deviations from the design specifications.

self-diagnostic

sequence-of-events recorder

A test of a component or system that is built-in to that component or system. A hardware device or software application that is used to provide records or logs of alarm and other event (e.g., actuation of a manual shutdown pushbutton) information.


37



shutdown interlock

site acceptance test

A device or group of devices arranged to sense a limit or offlimit condition or improper sequence of events and to shut down the offending or related piece of equipment, or to prevent proceeding in an improper sequence in order to avoid a hazardous event. Process of confirming performance of the total integrated ESD system to ensure its conformance to

TI

An abbreviation for test interval.

TMR

An abbreviation for triple modular redundant.

total time

The total time during which the ESD interlock should be able to respond to a demand.

triple modular redundant

A fault tolerant scheme that uses 2-out-of-3 (2oo3) voting to determine appropriate output action.

uptime

The amount of time that an ESD interlock is available to respond to a demand.

watchdog timer

A timer implemented to prevent the ESD system from looping endlessly, providing inaccurate communications, or becoming idle because of program errors or equipment faults.

written description

A method of describing the translation from a cause-andeffect matrix to an annotated logic diagram using textual statements.


38



ADDENDUM: DUANE PLOTS

In 1962, J. T. Duane postulated a mathematical formula to model software testing results. Once significant testing has been performed, and if meticulous records have been kept, the Duane Plot can show the status of software error detection. From this information the following can be determined: •

If progress is being made towards a stated reliability factor for the system.

•

A prediction of the testing time required until the next software error is found.

•

A prediction of the number of errors that will be found in a stated period of testing time.

•

A prediction of how many more hours of testing will be required to reach the desired reliability.

This method is valid as long as Types 1, 2 or 3 errors do exist, and they continue to be found and corrected. These error types are a measure of the severity of ESD program errors, such as Critical (Type 1), Major (Type 2), or Minor (Type 3). Figure 3 provides descriptions of these classifications. Some Type 1, 2, or 3 errors that occur during FAT or Pre-FAT ESD system software testing may be directly attributable to incorrect information supplied or communicated formally by Saudi Aramco. These errors shall not be used as Duane Plot data points, and they shall not be held accountable against the vendor. At the start of the FAT, the vendor must begin logging all errors encountered within vendor-developed logic and application programs in a software deficiency log, along with an error description, classification (i.e., Type 1, 2 or 3), proposed correction or corrective action, duration, and time encountered. This error logging must continue throughout the entire functional test period.


39



Actual FAT testing time must be used, (i.e., not calendar time or even CPU time). Based on previous experience, test hours should match the total number of man hours that the test team expended. The number of teams and the number of testing hours per day may vary during the ESD system test. However, the test time used should reflect the stress put on the ESD system during all tests as accurately as possible.

TYPE 1

A critical failure with disastrous effects, e.g., incorrect implementation of ESD logic, improper (Catastrophic) addressing of I/O points or bypass switch logic, software errors that contribute to one or more ESD output failures. Type 2 (Major)

TYPE 3 (Minor)

A failure that results in nonperformance of an ESD function or a degraded operation of the function, e.g., an error in I/O bypass logic that does not compromise the ESD system functionality, communication errors, mistakes in alarm settings, incorrect timer or counter presets, mistakes in ESD reset logic. ESD software errors that do not contribute to non-performance or a degradation of a required ESD function, e.g., errors in ESD program narrative, or comment files embedded within a program, errors in ESD documentation.

Figure 3. Types 1, 2, and 3 Error Descriptions

The collection of test data on a timely basis is essential to the construction and analysis of the Duane Plot. Because the plot is based on the number of errors discovered per testing time (in hours), the number of testing hours must be recorded on a daily basis, and the errors that are found must be promptly logged versus time.ofSaving a large number of errors to be reported at the end a longuptest makes the analysis more difficult or even impossible.


40



Error descriptions must be clear and specific, because they will be used later to classify and record the error data. Each discovered error in the ESD application program, logic or I/0 element addresses must be logged and itemized as a separate deficiency. Only in special cases (e.g., typographical errors in program narrative or editorial comments not pertaining to ESD logic) should a day's worth of errors be reported as one deficiency listing. Using the deficiency log, the vendor must construct a table of errors found versus the test time. The vendor must use this data to plot separate and unique "Duane Curves" for estimating the frequency of encountering future Type 1, 2 or 3 application program errors. The vendor must demonstrate from extrapolation of plot data that the following minimum probabilistic intervals of discovering future Type 1, 2 or 3 application program errors has been achieved: •

120 hours for Type 1 errors

•


•


The basic formula for the Duane Plot is: E/T = KTX Where: E = The sum of the errors occurring during time "T" T = Total testing time K = Constant X = Growth rate = Slope of the log-log plot of E/T versus T The formula holds as long as improvements continue to be made as a result of testing. Note that the equation is exponential in nature: E/T = KTX or E = KT(1+ X)


41



If test data are plotted linearly (total errors versus total time) the resultant plot approximates an exponential curve. The Duane Plot makes use of a log-log graph to allow easy determination of the slope of the graph. Calculate the "Accumulated Test Time" and the "Sum of the Errors Discovered" divided by the "Accumulated Test Time." Plot this data in log-log format. A look at the graphed data will reveal if any progress is being made towards improved reliability of the ESD system software. Progress is being made only when the slope of the curve is negative (i.e., when the number of errors found per hour of testing is decreasing). Further analysis of the data can provide additional insight into the reliability of the application program. The following are some examples that demonstrate the power of this method: Example 1: Differentiating the cumulative failure rate with respect to time gives the instantaneous or current failure rate. For example, one can use the first plotted point and the last plotted point to calculate the slope: SLOPE = X = Ln (last point E/T) - Ln (first point E/T) Ln (last point test hours) - Ln (first point test hours) Example 2: The number of additional hours of testing that will be required before the next error is likely to be found (Mean Time to Failure, MTTF) can be determined. In this case, the basic formula, E/T = KTX is solved for K. K = (E/T)/(TX)


42



Using cumulative values for E and T determined from the most recent point of ESD application program testing, the value of constant "K" can be solved. Once K has been determined, a prediction can be made in terms of the accumulated hours until the next ESDapplication program error occurs. Solving the basic equation for T: E = KT(X+l) T(X+l) = E/K [T(X+l)][l/(X+l)] = (E/K)[l/(X+l)] T= (E/K)[1/(X+l)] Time "T" represents the "Predicted" accumulated hours of testing necessary to locate the next ESD application program error. Example 3: An alternative and perhaps easier way to calculate the MTTF for ESD software (i.e., the next ESD application program error) is to use the differential of the cumulative failure rate with respect to time, as follows: E/T = KTX E = KT(I+X) Differentiating: dE/dT = (1+X)KTX Replacing KTX with its equivalent E/T: dE/dT = (I+X)(E/T) errors/hour MTTF = dT/dE = I/(I+X)(E/T) hours/error The above equation represents the hours of ESD software testing until the next error is discovered.


43



Figures 4 and 5 show Duane Plots that resulted from the FAT on two different ESD systems. In which system is progress being made toward a reliable software system? The slope of the curve in Figure 4 is negative, so progress is being made toward a more reliable system. The slope of the curve in Figure 5 is about zero, and no progress is being made toward a more reliable system.

LOG SCALE - TOTAL TEST HOURS S R U O H T S E T / S R O R R E E L A S C G O L

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

0 -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 -0.8 -0.9 -1.0

Figure 4. Duane Plot for ESD System 1


44



LOG SCALE - TOTAL TEST HOURS S R U O H T S E /T S R O R R E E L A C S G O L

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

0 -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 -0.8 -0.9 -1.0

Figure 5. Duane Plot for ESD System 2


45

Emergency Shutdown System Testing

Recommend Documents