Student Version

Auditing for Improvement

© Copyright ASQ 2012

2


Table of Contents

Auditing for Improvement Course Overview ........................................... 5

COURSE O VERVIEW AND INTRODUCTION .......................................................... 6 Overview .................................................................................................................. 6 Introduction to Auditing Fundamentals .................................................................... 6 Audit Preparation .................................................................................... 11

AUDIT PREPARATION MODULE GOALS ............................................................ 12 Module Overview.................................. ................ ................................... .................................. ................................... ................................ .............. 12 AUDIT S YSTEMS AND PROCESSES ....................................................................... 13 Audit Phases ................................. ................ .................................. ................................... ................................... ................................... ...................... 13 Phase 1: Audit Preparation ................................................................................... 14 Audit Preparation Phase, Step 1................................. ................ .................................. ................................... .......................... ........ 15 Audit Preparation Phase, Step 2................................. ................ .................................. ................................... .......................... ........ 16 Apply Auditing Rule #1 ................................... .................. ................................... ................................... .................................. ................. 20 Group Activity: Make Introductions ...................................................................... 21 Section Review ........................................................................................................ 21 AUDIT TEAMS ........................................................................................................ 22 Audit Preparation Phase, Step 3................................. ................ .................................. ................................... .......................... ........ 22 Auditing: Apply Rule #2.................................. ................. ................................... ................................... .................................. ................. 24 Auditing Group Activity: Team Formation.................................. ................. ................................... .......................... ........ 26 Audit Preparation Phase, Step 4................................. ................ .................................. ................................... .......................... ........ 27 AUDIT CRITERIA .................................................................................................... 28 Preparation Phase, Step 5 ...................................................................................... 28 The Third Rule of Auditing ................................................................................... 33 Group Activity: Audit Criteria .............................................................................. 34 DEFINE W ORKFLOWS ORKFLOWS ........................................................................................... 35 Preparation Phase, Step 6 ...................................................................................... 35 Group Activity: Workflows .................................................................................... 36 FORMAL NOTIFICATION....................................................................................... 37 Preparation Phase, Step 7 ...................................................................................... 37 Group Activity: Audit Planning ............................................................................ 39 S TUDY THE PROCESSES ........................................................................................ 40 Preparation Phase, Step 8 ...................................................................................... 40 Group Activity: Do the Turtle ............................................................................... 46 W ORK ORK P APERS........................................................................................................ 47 Preparation Phase, Step 9 ...................................................................................... 47 The Fourth Rule of Auditing ................................................................................. 50 Preparation Phase, Step 9 ...................................................................................... 51 Group Activity: Checklists ..................................................................................... 53 MODULE R EVIEW EVIEW : AUDIT PREPARATION PHASE ............................................. 54 Nine Steps of the Audit Preparation Phase .................................. ................. ................................... .......................... ........ 54 Four Products of the Audit Preparation Phase ........................................................ ........................................................ 54 Audit Preparation Summary of Key Concepts .................................. ................. ................................... ....................... ..... 55

© ASQ 2012

Auditing for Improvement Course Overview Audit Performance................................................................................... Performance...................................................................................57 57

AUDIT PERFORMANCE MODULE GOALS ........................................................... 58 Module Overview .................................. ................. ................................... ................................... .................................. ............................... .............. 58 AUDIT PERFORMANCE PHASE ............................................................................. 59 Phase 2: Audit Performance ................................................................................... 59 The Opening Meeting .................................. ................ ................................... ................................... ................................... ......................... ........ 61 Group Activity: The Opening Meeting .................................. ................. .................................. .................................. ................. 63 G ATHERING OBJECTIVE E VIDENCE .................................................................. 64 Phase 2: Audit Performance ................................................................................... 64 Gathering Objective Evidence ................................. ................ ................................... ................................... ............................... .............. 64 Tracing ................................................................................................................... 65 Interviews ................................................................................................................ ................................................................................................................ 66 Group Activity: Open-Ended Questions ................................................................. 69 Record Reviews ....................................................................................................... 69 Field Notes Examples ................................... ................. ................................... .................................. ................................... ....................... ..... 71 Group Activity: Gather Data ................................... .................. ................................... ................................... ............................ ........... 73 MODULE R EVIEW ................ ........................... .......... 74 EVIEW : AUDIT PERFORMANCE PHASE ................................. Audit Performance Summary of Key Concepts .................................. ................. ................................... ....................... ..... 74 Audit Reporting ....................................................................................... .......................................................................................75 75

W RITING RITING THE R EPORT EPORT .......................................................................................... 76 Phase 3: Audit Reporting ....................................................................................... 76 Audit Conclusions ................................... .................. .................................. ................................... ................................... ............................ ........... 76 Reporting Definitions .................................. ................ ................................... ................................... ................................... ......................... ........ 77 Create Audit Findings in Three Steps ................................... .................. .................................. .................................. ................. 78 Write the Summary ................................................................................................ 83 Group Activity: Write a Finding ................................. ................ ................................... ................................... ......................... ........ 84 THE AUDIT CLOSING MEETING ......................................................................... 85 The Closing Meeting ............................................................................................... 85 AUDIT R EPORTS EPORTS AND R ECORDS ECORDS .......................................................................... 86 The Audit Report ................................. ................ .................................. ................................... ................................... ............................... .............. 86 Group Activity: Write a Summary ......................................................................... 86 Compiling and Distributing Audit Reports ................................. ............... ................................... ............................ ........... 87 Audit Records ................................... .................. ................................... ................................... ................................... .................................. ................ 88 MODULE R EVIEW EVIEW : AUDIT R EPORTING EPORTING PHASE ................................................. 89 Summary of Key Audit Report Concepts .................................. ................ ................................... ............................... .............. 89 Audit Closure ........................................................................................... 91

CORRECTIVE ACTION ........................................................................................... 92 Group Activity: Respond to a CAR .................................. ................. ................................... ................................... ..................... 95 Evaluate and Verify the Implementation.................................. ................ ................................... ............................... .............. 96 Auditing for Improvement Course Summary ........................................ ........................................97 97

COURSE R EVIEW EVIEW .................................................................................................... 98 Phases of the Audit ................................................................................................ 98 Value of the Audit .................................. ................. .................................. ................................... ................................... ............................ ........... 98 Rules for the Auditor and the Audit .................................. ................. ................................... ................................... ..................... 99 Audit Results ................................. ................ ................................... ................................... ................................... ................................... .................100 100 Instructor Contact Information ..............................................................................100 © ASQ 2012

3

4


© ASQ 2012

Auditing for Improvement Course Overview


© ASQ 2012

5

6


Course Overview and Introduction

Overview Instructor Section Objectives Introduce yourself

§

§

§

§

§

Special thanks to Dennis Arter for providing content for this ASQ course.

Orient attendees to manual Review class agenda and management Explain Handout 1 Use PowerPoint to introduce auditing. Check attendee understanding.

Section Duration 1 hour §

•

•

•

•

•

•

Define products and processes and systems Study processes to identify information needs Gather information and analyze it Present the conclusions in a way that focuses on system issues Do it all in sector-specific teams Have fun and learn

First, one must define the things to be audited. Then one must understand how those things are done. Now the auditor is in a position to harvest data and make something of it. Finally, the results must be meaningful to all interested parties.

Materials Used Manuals (attendee, instructor) §

§

PowerPoint presentation, with detailed instructor notes in slide notes

Instructor Demonstrates Book: Quality Audits for How to usePerformance, Handout 1 by Improved § §

§

DennisModel Arteruse Audit

Introduction to Auditing Fundamentals After completing this section, you will: •

•

•

•

Understand the history and general model of auditing Understand the position and importance of the audit coordinator Understand the relationship between audit and inspection Recognize the rules and phases of an audit

In order to: •

•

•

Ensure that an audit coordinator is designated Assist in developing an annual audit schedule Help write an internal audit procedure

© ASQ 2012


Control era focused on product. Inspection was the primary tool. It began as we moved away from agriculture and into manufacturing (about 1920) with some of the early pioneers. It lasted about 50 years. Assurance era started shortly after World War II, as military and nuclear industries emerged, with powerful and dangerous technologies. We focused on defining and doing processes in a uniform fashion. It lasted about 25 years. Management era started as the 20th century was ending. ISO 9001:2000 was its grand entrance. We began to understand and control systems. It will probably last for another 12 to 13 years. Integration era is just beginning. All management control systems are a form of PDCA and support risk management. It will probably last another six years. Notice the exponential length of each (half the time of the previous). This ties into the principle of accelerating change.

© ASQ 2012

7

8

Auditing for Improvement There are five main components to an audit.

Instructor Note Discuss the five components of an audit. §

§

Point out that the Requirements box represents what organizations are supposed to do vs. the Evidence box, which represents what organizations actually do.

The Requirements box on the top left side of the diagram represents what organizations are supposed to do. The Evidence box on the top right side represents what organizations actually do. During inspection, auditors compare these two sides. 1. All audits start with conforming and nonconforming data. 2. Auditors then analyze the data. Those become findings. 3. Auditors further analyze all the findings to draw conclusions. They are required to form opinions.

© ASQ 2012


9

This diagram represents a more detailed picture of the audit model.

Officially called an “audit program manager” •

•

•

© ASQ 2012

Champion of the audit to senior management Responsible for audit resources and contributions: o Team members and training Budget and quarterly reports to management o Responsible for the audit schedule: Not all controls are equal—use cost, production, risk o Schedule should reflect production activities o Notify those on the schedule o

Instructor Note Ask: How many have an audit coordinator? What’s the definition? Refer those who don’t to Handout 1. §

10


The audit schedule should be shared with everyone on the schedule. The o nly time we do surprise audits is when we suspect deliberate waste, fraud, or abuse (which is uncommon). Sharing the schedule with everyone on the schedule is an example of extending a hand in friendship (the first of the Five Fingers of Friendship) to build trust between both parties. Four more examples, one for each remaining finger, will come later in the course.

Instructor Note Invite attendees to respond. Check for understanding. Observe body language, notetaking, or review/remind. Provide remediation as needed. §

Progress Check 1. Name the five components of an audit. 2. Describe the role of the audit coordinator. 3. In which era of management do we live?

© ASQ 2012

Audit Preparation

Audit Preparation

© ASQ 2012

11

12


Audit Preparation Module Goals Module Overview After completing the audit preparation module, you will: •

•

•

Understand audit systems and processes Build a successful audit team based on selection guidelines In teams, develop audit plan criteria and conduct initial evaluations that help prepare you for the audit

© ASQ 2012

13

Audit Preparation

Audit Systems and Processes In this section, you will learn to: •

•

•

•

•

•

§

Recognize the nine steps used to prepare for an audit Understand the application of performance improvement audits Understand an organization’s three basic process types Understand how processes make a system Define the audit’s purpose and scope Apply the first of four auditing rules

Ensure attendees meet all section learning objectives


In order to: •

•

Reduce waste and improve efficiencies of local (your own) processes Improve and increase communications between functional departments

Materials Used Manuals (attendee, instructor), Handout 1 §

Audit Phases

Effort Required by Audit Phase 10% Phase 4: Closure

PowerPoint presentation

§

Chart pad/white board

§

Markers

Instructor Demonstrates How to define the audit purpose and scope §

25% Phase 1: Preparation

15% Phase 3: Reporting

§

§

How to provide useful audit information

50% Phase 2: Performance

Instructor Note Briefly review the chart. Convey that phases will be explained in greater detail. §

© ASQ 2012

14


Phase 1: Audit Preparation §

Review the nine steps of audit preparation phase with the learners.

Complete these nine steps:

All nine steps are necessary before you can start doing fieldwork.

Note

© ASQ 2012

15

Audit Preparation

Audit Preparation Phase, Step 1

Why are you doing this? •

•

•

Routine monitoring Special problem resolution Preparation for change The purpose of an audit must be defined during audit pl an development. It does not remain constant throughout the year.

Note

© ASQ 2012

Audits are standard management practice. Problems are the exception, not the rule.

Instructor Note Solicit examples for each bullet. Provide examples as needed. §

16



Instructor Note Ask: What’s the difference between audit and inspection scope? Answer: Audit scope is always different. Doing the same audit over is inspection. §

§

Give an example for each bullet (see slide notes for examples, if needed).

What will you look at? •

•

•

•

Products Processes Functional areas Time The scope of an audit will always be different. Doing the same audit over and over is just inspection.

Note

This is why the audit scope is generally limited to the past year’s work, or since the date the area was last audited. Audits are performed in pieces, not in one big “meal.”

Product: different product lines, such as high-pressure steam valve and low-pressure water valve. Processes: recordkeeping processes, design review processes.

For Example

Functional areas: maintenance, purchasing, warehousing. Time: audits anticipate the future (six months to one year from the present time).

© ASQ 2012

17

Audit Preparation

Instructor Note Tell students: Every organization (business, government, nonprofit, or NGO) uses these three kinds of processes. As auditors, you need to examine them all.

Every organization should have three processes audited:

§

Instructor Note Point out that these are all action verbs, ending in -ing. §

Production processes make goods and services you provide. For example: Assembling

Fastening

Operating

Serving

Coating

Growing

Quilting

Teaching

Cooking

Inspecting

Riding

Testing

Deicing

Moving

Sequencing

Washing

§

© ASQ 2012

Point out that these are typical support processes.

18


Strong processes facilitate production. For example:

•

•

•

•

§

Equipment calibration Equipment and facility maintenance Quality, environment, safety, security Accounting, information technology

•

•

•

Purchasing, human resources, training Production planning (e.g., shift schedules) Research

Ask: Are you familiar with COPs and SOPs? Give examples as needed. (see slide notes for examples)

External interface processes are customer-oriented (COP) or supplieroriented (SOP). They affect:

•

•

•

Marketing Sales Customer Support

•

•

•

Shipping Regulatory Affairs Purchasing

This is a small part of audit planning, typically developed through experience.

Note

Interface processes are special due to the difficulties in moving clear and correct information across organizational boundaries. They receive 80 percent of what you send. Communications channels are not perfect.

© ASQ 2012

19

Audit Preparation Systems are linked processes

System: Processes working together to achieve a common goal.

Explain how systems are linked processes generating multiple Instructor inputs andNote outputs. Illustrate how change moves through a system. §

A system is a complex network of processes Audits ask: Do all processes connect smoothly?

§

§

Audits ask: Do all processes connect smoothly?

© ASQ 2012

Add to Handout 1, Column A: Peter Senge’s book The Fifth Discipline.

20

§

Ask: Why do we do audits? Participants will apply these questions to the audits they perform.


At a macro level, audits examine systems and their implementing processes Auditors want to know: •

•

•

Are these systems achieving their business objectives? Are these systems stable? Do all the processes connect smoothly? Audits look at the future. Inspections look at the past.

Note

Apply Auditing Rule #1 Instructor Note Ask: What sort of connotations does an audit have? Give examples. §

This is the first of four fundamental rules. Auditors m ust follow the four rules, no matter what. Most audits dwell on the past. Auditors must change this thinking to look at the future so that stakeholders can make intelligent, forwardthinking decisions that help the company do better. That’s the true purpose of an audit.

© ASQ 2012

21

Audit Preparation

Group Activity: Make Introductions Make Introductions •

•

•

Who are you? What does your firm make? What involvement do you have in auditing?

Instructor Note Answer these questions. Then invite attendees to answer. §

§

On chart pad/white board, write each attendee’s first name and business sector (helps form teams later).

Section Review You should now be able to: •

•

•

•

•

•

© ASQ 2012

Recognize the nine steps used to prepare for an audit Understand the application of performance improvement audits Understand an organization’s three basic process types Understand how processes make a system Define an audit’s purpose and scope Apply the first of four auditing rules

Instructor Note Invite and answer questions. Check for understanding. Offer remediation as needed. §

22


Audit Teams


Instructor Section Objectives Ensure attendees meet all section learning objectives

§

In this section, you will: •

•

Section Duration 30 minutes §

•

•


§


In order to: •

•

Instructor Demonstrates How to select an audit team §

§

Identify employees with auditor potential Improve communication and cooperation among functional departments Reduce audit burden on individual staff members Know how to achieve internal and external recognition (credentials) of auditors

•

•

Select qualified audit team members and evaluating teams for effectiveness Apply the second of four auditing rules Meet the three conditions necessary for auditor qualification Identify certification options for auditors

How to improve communication among departments

© ASQ 2012

Audit Preparation

Audit Team FAQs

Who’s on a team? One or more qualified auditors What’s the average team size? Two to six people

What’s the average duration? One to five days

The Audit Coordinator selects and qualifies a small nu mber of team leaders. Generally, these team leadership positions are part-time roles (three to four audits per year are typically conducted). Qualification papers are renewed annually. Afterward, team leaders solicit assistants and qualify them (they may never have audited before). Qualification papers are used only for the audit currently in progress. This promotes diversity, spreads information, breaks down interdepartmental silos, and promotes the value of auditing at a personal level. Subject matter experts can be counseled on ways to best help direct inquires and interpret data. They must not, however, provide direction to the auditee. The duration listed is applicable to both internal and supplier audits. © ASQ 2012

23

24


Auditing: Apply Rule #2 To best define the team, apply the second rule of auditing.

Instructor Note Questions to pose when identifying auditor qualifications §

§

Describe relationships between team leader, auditors, and subject matter experts

How do we know the audit team is qualified?

•

Objective and unbiased

Technical knowledge of processes •

Knowledge of audit practices •

© ASQ 2012

25

Audit Preparation •

•

•

Note

Auditors are not independent In terms of the PDCA model, someone else has done the P (Plan) and the D (Do); auditors do the C (Check) Auditors don’t own what they’re auditing.

Auditor Skills

Types of qualification “papers” I. Local requirements o o

Your own standards Communicator, auditing, technical

II. Professional society o

ASQ Certified Quality Auditor

III. National and international schemes o o

© ASQ 2012

Conformity assessment Accreditation

Instructor Note Define the characteristics of a good auditor and skills acquired §

26


All auditors should have “papers.” Auditees will (and should) ask to see them.

Note

Instructor Note Define types of “papers” and discuss merits of CQA. §

§

§

Refer to Dennis Arter’s book.

Solicit responses from the group. Remediate as needed.

•

•

•

Why are teams better than single auditors? What are the three components of qualification? How can we prove auditor competence?

Auditing Group Activity: Team Formation Instructor Note Discuss effective team-building skills §

§

Evaluate teams for criteria achievement

1. Based upon your business sectors, form teams of two to six people. 2. Introduce yourselves again. 3. Choose one of your organizations and perform the first steps of an internal audit. o o

§

What does the organization provide? Who are its customers?

10-minute break

© ASQ 2012

Audit Preparation


Do you have permission?

© ASQ 2012

27

28


Audit Criteria

Preparation Phase, Step 5

Instructor Section Objectives Ensure attendees meet all section learning objectives.

§

After completing this section, you will: •

•

Section Duration 30 minutes

•

§

•


§


Identify audit requirements Know where to find external and internal requirements for operations Know the difference between a document and a record, and the different document forms Understand the importance of early communication with the auditee

In order to: •

•

Improve internal formal (written) controls Reduce auditor/auditee misunderstandings

Instructor Demonstrates How to identify audit requirements. §

§

How to determine which document to use.

© ASQ 2012

29

Audit Preparation

Fundamental Principles: •

•

•

•

Audits measure to performance standards Auditors cannot “own” the requirements Requirements are known as the “basis” for the audit Requirements must be defined and accepted by all parties before they can be audited

This is the major difference between auditing and consulting. In consulting, the requirements have not yet been defined. Consultants help to define these. In auditing, the requirements are there.

Instructor Note Differentiate between consulting and auditing. §

§

Third rule of auditing aud iting will be discussed later.

Note

Instructor Note Point out that requirements are on the left side of the model. §

§

© ASQ 2012

External = external documents

30


Audit Requirements are the PDCA cycle’s plans

Instructor Note Explain the PDCA cycle. §

§

§

Requirements fall under the Plan component.

We must have requirements to audit against. Requirements are the “P” (Plan) of the PDCA (Plan, Do, Check, Act) cycle. Auditors must have some activity to look at. The audit is part of the measuring box, or the “C” of PDCA.

Auditing falls under the Check component.

© ASQ 2012

31

Audit Preparation

Documents

Requirements are captured in documents . (Remember, a document is before, and a record is after. Many people do not realize this distinction.) Here is the classic document pyramid, but modified to make certain aspects clearer.

Instructor Note Describe the document pyramid. §

§

External documents come from distant sources: government regulations, international standards, corporate policy, etc. They are generally quite highlevel, strategy-driven, and outcome-based.

The manual (normal term) is site-specific and states how the local organization will meet the external requirements. Manuals are supposed to be skinny. Their greatest benefit comes from the initial writing. They are frameworks to accomplish the strategies of the organization. Procedures are process-focused. They cover actions within the organization. Procedures are job performance aids for an already-trained and qualified individual . They are not used to substitute for training and qualification by the supervisors. Best procedures are no more than six pages. Specifications are product requirements: formulas, quantities, form-fitfunction, drawings, assembly sheets, etc. They used to be called work instructions, but that caused confusion. Notice the system-process-product relationship: It appears many times in an audit.

The auditee has control over everything internal. A hierarchy exists.

© ASQ 2012

Explain the system-processproduct relationship.

32

Instructor Note Some examples of external documents. Students may provide additional examples. §

Instructor Note Some examples of internal documents. Students may provide additional examples. §


External documents are not produced by the auditee. They include:

National regulations

Package inserts

International regulations (export)

Material safety data sheets

National and international standards

Corporate policies

Industry codes and standards

Customer requirements

Internal documents are produced by the auditee. They include: Functional manuals (quality, research, production, accounting, etc.) Departmental procedures (administrative) Standard operating procedures Production planning sheets Inspection plans Software settings These external and internal examples apply to all firms and business sectors.

Note

Instructor Note Inform class that documents are not always printed anymore. §

Documents may come in these forms: •

•

•

•

•

•

•

•

Instructor Note Discuss second (of 5) Finger of Friendship: Informal Contact. §

Paper versions of manuals and procedures Computer files (word, database, drawing) Hyperlink documents (html, xml, .net) Pictures and samples Memoranda and emails Informal (post-it notes, marks, pocket notes) Vertical: all controls in one unit Horizontal: one control in all units

Make a “request list” that includes: •

•

All external documents High-level internal documents

© ASQ 2012

Audit Preparation

Team leader should contact the auditee: •

•

•

“Informal contact” Expect fearful initial reaction Expect reluctance to open up

The Audit Coordinator and the Second Finger of Friendship •

•

•

Since the audit coordinator doesn’t know what all these documents are, and the buyer doesn’t know, the team leader needs to informally contact the auditee to get this information directly. This is also the time to get copies (or file locations) for study. This informal contact improves information flow and builds tr ust. Informal contact is the second of five fingers of friendship.

The Third Rule of Auditing To align expectations with results, apply the third rule of auditing:

© ASQ 2012

33

34


Group Activity: Audit Criteria Instructor Note Refer students to Handout 2. Ask them to create a list of the following required documents to share with the class in 10 minutes. §

With your team, please make a list of required documents: Three external documents Three high-level documents Three process documents Three task or specification documents Share with the class in 10 minutes. •

•

•

•

© ASQ 2012

35

Audit Preparation

Define Workflows

Preparation Phase, Step 6 Instructor Section Objectives Ensure attendees meet all section learning objectives.

§



§


§

Chart paper

§

Marker

Instructor Demonstrates How to draw a flowchart §


Recognize and draw simple flowcharts of audited activities

In order to: •

•

•

Improve internal communication and understanding Identify processes in need of greater definition Reduce operating risk and improve customer satisfaction

For internal and external audits, understand what’s being made: •

•

•

•

Read procedures Converse with a trusted friend Reference prior audit files Consult with your contact person via telephone, email, and websites

Instructor Note Advise students to avoid site visits and to be technically competent by studying procedures, process files, product descriptions, and audit files. §

§

In order to: •

© ASQ 2012

Make a flowchart

Emphasize that output is a flowchart.

36


Example: How to Make French Fries

All auditors must first be able to flowchart what they will be auditing. Create a simple, one-page flowchart. Use existing flowcharts as sources, but develop them on your own.

Group Activity: Workflows Instructor Note Ask students to choose an activity to audit and flowchart. Complete within 20 minutes. §

§

§

Check to ensure verb-object for each box (total of eight) is completed accurately.

With your team, choose an activity to audit and flowchart it: 1. 2. 3. 4.

Focus on production, not paperwork and inspection. Use a maximum of eight boxes. Use a maximum of four words per box. One word must be a verb. Below each box, identify who performs the task.

Complete within 20 minutes.

10-minute break

© ASQ 2012

37

Audit Preparation

Formal Notification

Preparation Phase, Step 7 Instructor Section Objectives Ensure attendees meet all section learning objectives

§


•

Apply positive communication methods Develop an audit plan

In order to •

•

•

© ASQ 2012

Improve internal communication and understanding Reduce misunderstandings between auditor and auditee Build trust


§


38


Instructor Demonstrates How to develop an audit plan.


Step 7: Notify the Auditee

Audit Plan Components

§

§

How to communicate effectively.

1. Company or group audited

5. Requirement documents

2. Audit purpose

6. Interfaces with others

3. Audit scope

7. Team members

4. Sections/sub-groups affected

8. Overall schedule

Prepared by: Team leader Reviewed by: Team members Approved by: Audit coordinator

Instructor Note Review these audit plan components.

These plans are different from the annual audit schedule.

§

§

Each audit must have a one-page plan, developed by the team and approved by the audit coordinator.

Tell students they will create their own audit plan later.

Note

Instructor Note Sending the audit plan to the auditee is an expected practice. This may be different from what students are used to. If so, have them add it to Handout 1. §

§

Emphasize that auditors should plan for challenges with advanced contact and allow plenty of time.

Share Audit Plan (Formal Notification) •

•

•

Performed by the audit coordinator There should be no secrets or surprises Not looking for waste, fraud, or abuse o o Objective: build trust and cooperation Desirable goal: contact 30 days before the fieldwork starts

Send the audit plan to the auditee. This is expected practice. This may be different from what you are used to doing (refer to Handout 1). Advanced contact requires planning. This may be challenging.

© ASQ 2012

39

Audit Preparation

Group Activity: Audit Planning With your team, develop an audit plan 1. Cover two days over several shifts 2. Include both external and internal requirement documents (from previous workshop), but make up specific names

Complete in 15 minutes

© ASQ 2012

Instructor Note Advise students to use Handout 3. §

§

§

Complete activity within 15 minutes. Check teams work for understanding.

40


Study the Processes Preparation Phase, Step 8


§

Section Duration 30 minutes


•

§

•


§


§

Handout 4

Understand the simple (four-box) process model’s strengths and limitations Know the six universal process affecters and how they influence processes Create turtle diagrams for audited processes

In order to: •

•

Increase knowledge of internal processes Understand operations holistically

Instructor Demonstrates How to create turtle diagrams. §

© ASQ 2012

41

Audit Preparation

Universal “Four-Box” Process Model

Inputs and outputs are always different and always measurable. They have form-fit-function. Resources allow the process to go, and controls keep it stable.

Instructor Note Ask a random student: “Why is it called a four-box model?” §

§

This model (which first appeared in the 1990s) is the genesis of SIPOC (supplier-input-process-output-customer) analyses. It’s a good high-level analytical tool, but you cannot use it to study processes at the level needed for good auditing.

© ASQ 2012

Explain that four-box models are best used for high-level analytics, not auditing.

42

§

§


Example from the pharmaceutical industry

Example from a local bank

© ASQ 2012

43

Audit Preparation

§

Is there a better way? •

•

Simple four-box approach requires significant concentration. What about tapping previous work on how processes behave (such as Ishikawa)? We often don’t capture everything with the four-box model. It’s best used for high-level analytics.

Note

© ASQ 2012

Example from a clinical lab

Instructor Note Ask students to answer the second bulleted question. §

44


Universal Process Affecters

Instructor Note Point out to learners the mnemonic device: MMMMME. §

Instructor Note Remind learners that environment does not mean “green” and that manpower includes womanpower. §

Whoa! This method gives us too much detail! How about something halfway?

© ASQ 2012

45

Audit Preparation

If you perform this universal process affecters analysis method on each of the boxes in the process map, it will take far too long. Note Turtle Diagram

§

The Turtle Diagram is a human-friendly way to get the brain to think about things that are important to the process about to be part of the audit. The concepts of the four-box analysis and concept of the universal process affecters analysis are applied to this method of analysis. Turtle: process = body, inputs = head, outputs = tail, four boxes = legs.

© ASQ 2012

Point out where each concept, four-box analysis, and universal process affecters analysis are integrated in this diagram.

46


Instructor Note The modified turtle diagram adds environment and safety. These additions are important because quality professionals are now being asked to analyze safety and sustainability. §

The modified turtle diagram adds environment and safety. These additions are important because more and more quality professionals are being asked to analyze safety and sustainability.

Instructor Note Refer the students to Handout 4.

Group Activity: Do the Turtle

§

§

§

Check teams for understanding. Complete within 20 minutes.

1. Choose a process from your team’s flowchart. 2. Start a turtle diagram for your process (two items per box). 3. Share your diagram with your team members. Brainstorm additional items.


§

10-minute break

© ASQ 2012

47

Audit Preparation

Work Papers Preparation Phase, Step 9 Instructor Note Point out that work papers are also commonly called checklists. §

Work papers are also commonly called checklists

Instructor Section Objectives Ensure attendees meet all section learning objectives

§

Note


•

•

•

Know the five kinds of objective evidence Understand the difference between audit checklist and interview questions Know the characteristics of a good checklist question Understand how checklists will improve the value of gathered data



§


§

Handout 5

In order to: •

•

© ASQ 2012

Increase your efficiency as an auditor Ask appropriate questions

Instructor Demonstrates How to write a good checklist. §

48


Instructor Note Instruct students to write down DDD, which stands for Data Drives Decisions.

Auditors need facts. Work papers assist us in getting those facts. This is the fourth and final rule of auditing: DDD: Data Drives Decisions.

§

§

Inform students that this is the foundation of the Baldrige approach.

Note

•

•

•

Data Drives Decisions Audits require facts (data) to form conclusions that drive decisions Work papers gather facts

General Model for Auditing

Step 9 is the top right side of the audit model.

Note

© ASQ 2012

49

Audit Preparation

Five Forms of Objective Evidence

Physical – things you can touch, count, and see. Examples: dimensions and quantities. Sensory – primarily sight, smell, and hearing. These are different from physical in that you don’t use a micrometer—you use your own measuring devices. Ask yourself, “would you use your mouth?” Example: aluminum. Paperwork – a document tell us what we’re going to do. Records tell us what we did. Interviews – the most important source of info we’ve got. Examples: operators, techs, analyzers, engineers, call center representatives, buyers, and managing directors (we’ll revisit interviews later on in the course). Patterns – percentages, trends, and ratios of the above four forms of evidence.

These bulleted points above are desirable characteristics of a checklist question.

Note Objective evidence should: •

•

•

•

© ASQ 2012

Come from many different documents and levels (tiers) Contain one fact per question (single issue) Be objective (no “fuzzy” words) State the source of the requirement

50

Instructor Note Ask students to answer each bulleted question and provide remediation. §

§

Remember the top of the pyramid. The only way to dig deeper into the organization is to customize.


Issues to be explored: Should we use standard checklists? Should we show them to the auditee? Do we have to stick to them? Detailed checklists are the most common form of work papers. •

•

•

Sharing a customized checklist continues to build the trust between both parties (the fourth of the Five Fingers of Friendship). Share one week before the audit starts. Checklists (the most common form of work papers) must be very detailed to get sufficient and objective facts. Having 50 to 60 pages per audit is quite common.

The Fourth Rule of Auditing To serve the needs of your stakeholders, apply the fourth rule of auditing:

© ASQ 2012

51

Audit Preparation

Avoid this pitfall: Opinions are often found in conclusions, and are typically rejected.

Tip

§

10-minute break

§

Explain the turtle diagram.

Preparation Phase, Step 9 Contrast:

The difference between checklist and interviews: always write down checklists; never write down interviews.

Tip

The turtle diagram gets us to read the requirements documents, which gets us to read the controls and turn them into checklists so we can identify the objective evidence.

© ASQ 2012

52

Instructor Note Point out the checklist example of bulleted questions. §

§

Define information needs: turtle “what”

Mention that a good checklist leads to good objective evidence.

Instructor Note This is another example of how the turtle diagram assists with writing a checklist. §


Define information needs: turtle “with whom”

© ASQ 2012

53

Audit Preparation

Group Activity: Checklists Using the checklist handout and your team’s experience: 1. Invent two (or more) documents for each turtle box. 2. Invent two or more requirements for each document. 3. Write a checklist question for each requirement.

Instructor Note Refer students to Handout 5. §

§

Complete within 60 minutes. §

© ASQ 2012

Instruct students to start with two checklist questions for each turtle box. Share and critique, then write very specific questions. Spot-check and critique students’ work during the entire 60-minute activity.

54


Module Review: Audit Preparation Phase Nine Steps of the Audit Preparation Phase Instructor Note Briefly summarize all nine steps. §

Complete these nine steps:

Four Products of the Audit Preparation Phase Instructor Note Review the four products of the preparation phase. §

§

Instruct students to develop an audit plan and checklist.

Four products of the preparation phase •

•

•

•

Audit plan Initial study and evaluation of documents Work papers Plan for action

© ASQ 2012

55

Audit Preparation

Audit Preparation Summary of Key Concepts What you’ve learned •

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

History of auditing General model of auditing Audit coordinator Nine steps of preparation Three types of organizational processes How processes make a system Three criteria for auditor qualification Make-up of the audit team Difference between document and record Levels and forms of documents Power of flowcharts Audit plans Four-box process model Six universal process affecters Turtle diagrams Five kind of facts Checklist vs. interview Turtles to checklists

§

© ASQ 2012

Remind students about tomorrow’s start and stop times, place, and any other logistics.

56


© ASQ 2012

Audit Performance

Audit Performance

© ASQ 2012

57

58


Audit Performance Module Goals

Module Overview In this module, you will: Practice clarifying and understanding important audit performance concepts In order to: •

•

Increase your confidence performing fieldwork

© ASQ 2012

59

Audit Performance

Audit Performance Phase

Phase 2: Audit Performance §


•

•

•

•

•

Identify six key components to successful audit performance Become familiar with action items and schedules for one- and two-day audits Recognize how opening meetings can improve your relationship with auditees Understand meeting mechanics and topics Prepare an opening meeting agenda and an audit agenda Determine effective methods for communicating progress and issues to management

§

Before beginning Day 2’s new course material, briefly review the key concepts presented during Day 1. Invite students to ask and answer questions related to Day 1 material.


§


In order to: •

Reduce tension between auditors and auditees

Key components to successful audit performance

© ASQ 2012


§


Instructor Demonstrates How to successfully perform an audit. §

60

§

Point out an example of oneday audit, although it is not recommended due to heavy workload.


Here’s an example of a typical one-day audit: Morning

Afternoon

8:00-8:15 a.m. Opening meeting 8:15-11:30 a.m. Data gathering 11:30 a.m.-Noon Team meeting Noon-12:30 Lunch

12:30-3:30 p.m. Data gathering 3:30-4:30 p.m. Team meeting 4:30-4:45 p.m. Close meeting

A two-day audit is recommended over a one-day audit. Here’s an example of a typical two-day audit: Day 1

Day 2

8:00-8:15 a.m. Opening meeting

8:00 a.m.-Noon Gathering Data

8:15-11:30 a.m. Data gathering

Noon-12:30 p.m. Lunch

11:30 a.m.-Noon Team meeting

12:30-2:30 p.m. Team meeting

Noon-12:30 p.m. Lunch

2:30-2:45 p.m. Closing meeting

12:30-3:30 p.m. Data gathering 3:30-4:30 p.m. Team meeting 4:30-4:45 p.m. Daily briefing

This schedule for the day also applies to supplier audits. Supplier audits often include additional teamwork in the evening. Note

© ASQ 2012

61

Audit Performance

The Opening Meeting §

The opening meeting: Continues auditor/auditee communication Is required for all audits Your meeting action items: •

•

Distribute checklists and other work papers Define the detailed schedule for the rest of the audit

•

•

The team leader brings four copies of the (customized) checklist. Of the four copies, keep two copies, one copy goes to a senior person on their side (out of respect), and one copy goes to your guide/escort, if you have one. •

Note

The most important outcome of the opening meeting is the detailed schedule for the rest of the audit.

Present your schedule template

Prepare the audit schedule ahead of time and bring along several copies.

•

© ASQ 2012

Tip

§

Explain the definition of an escort or guide. Mention the primary duty of guide is safety.

62


Together, complete the schedule.

§

Review the five fingers of friendship: 1. Share audit schedule 2. Informal content 3. Copy of audit plan 4. Detailed checklist questions in advance 5. Collaborate on the conduct of the audit

At the meeting, fill in the details all together. This allows you to respond to last-minute difficulties and be flexible. This also allows everyone to influence how you perform the audit.

• §

Mention you should complete the schedule with auditees.

Tip

Using the above example, you may learn that Tom Campbell in the Laboratory department has a representative coming into the lab on Tuesday. You’ll learn this at the meeting, and can help him out by rescheduling to Wednesday.

Meeting Tips 1. 2. 3. 4. §

Keep the meeting low-key. Minimize attendance: six to eight people, including auditors. Keep it brief: 10 to 20 minutes. Take charge: Pass out a meeting agenda.

Provide examples that demonstrate how to keep the meeting low-key and brief.

© ASQ 2012

63

Audit Performance

Group Activity: The Opening Meeting Based on what you’ve just learned, develop some details for your team’s opening meeting. Use Handout 6. Complete within 15 minutes.

§

Refer students to Handout 6.

§

Monitor student progress.

§

Invite questions.

§

§

© ASQ 2012

Assess understanding, and review any concepts that may need additional attention. Discuss/suggest approaches to keeping managers informed of audit progress and concerns.

64


Gathering Objective Evidence Phase 2: Audit Performance In this section, you will: •

•

•

•

•

•

•


§

In order to: •

•



§


§

Chart sheet

§

Marker

Instructor Demonstrates How to conduct an interview. §

§

§

Discuss the process of gathering objective evidence.

Understand tracing and its application to auditing Understand the need for corroboration and three common methods Know the six steps for conducting an interview Understand the importance of no secrets to all parties Know common protocols for examining records Know common protocols for observing activities Know what objective evidence statements look like

•

•

Use an interview sheet on the job Avoid misunderstandings between auditor and auditee Get an honest picture of existing practices Gain confidence in ability to conduct an interview

Gathering Objective Evidence What it is: •

The fieldwork conducted after the opening meeting

How to gather: •

•

•

Walk the processes Interview the people performing the tasks Go to them

Read the processes during your preparation phase and ask students pertinent questions.

Split up the audit team and start collecting data. Visiting auditees at their job site will give you a much more detailed picture of the work being performed. Tip

Reading the processes during your preparation phase will help you walk the processes and ask the most pertinent questions during this phase.

© ASQ 2012

65

Audit Performance

§ •

•

Need both types: conforming (“good”) and nonconforming (“bad”) Recall the five kinds of data §

§

Remind the learners about the five kinds of data found in the general model for auditing. Provide examples of good and bad evidence. Solicit learners for examples of each of the five kinds of evidence and each of the two types of evidence.

Tracing §

§

© ASQ 2012

Explain when and where to apply the MMMMME Universal Process Affecters

Demonstrate questioning using the principles of probing for each department listed.

66


The MMMMME Universal Process Affecters are applied to audit tracing. Tracing is the most common method used. Backwards tracing is usually easier and more valuable because you work backwards from what has already been done instead of working forward while speculating when it will be done. Note

§

10-minute break

Interviews •

•

•

§

Use your checklist and principles of probing to collect data (this is why you must understand the processes before you leave for the field).

Difficult to conduct/become easier with practice Must be corroborated o Another source o Another receiver o Another object Use open-ended questions o Who? What? When? Where? Why? How? Would you show me?

Provide examples of each type of corroboration.

Because of natural human tension (and personal space), exchange of information is always less than perfect. You don’t hear everything or always say what you mean. Each person has approximately 30" of personal space around them (changes with culture). Note

Tools for communication: corroboration and open-ended questions.

© ASQ 2012

67

Audit Performance

§

§

© ASQ 2012

Suggest discussing something neutral to break the ice (more on this in Arter’s book). Demonstrate how to use the paperwork to neutralize tension. Remind students to ask for it back.

68


Step 3: Emphasize open-ended questions, not checklist questions. Questioning examples:

Closed, checklist question: “Do you have a method for identifying nonconforming items?” Answer: “Yes.” Tip

Open-ended question: “What is the method for identifying nonconforming items?” Answer: “(auditee’s specific answer).” Step 4: When noting responses, actively listen to the auditee and repeat back what was just heard. Paraphrasing and “write out loud” are good analyzing tools.

§

§

§

Emphasize open-ended questions, not checklist questions. Discuss good analyzing tools such as paraphrasing and writing out loud.

Emphasize the principle of no secrets as a win-win outcome. © ASQ 2012

69

Audit Performance

Auditors are now allowed to say good things, as accepted and encouraged practice (this has not always been the case). Including positive observations of good practices is beneficial to building strong relationships. Note

If you suspect something is wrong, you are obligated to inform the auditee that the audit will not pass. For example, “It looks like we may have a problem here.”

Group Activity: Open-Ended Questions Review Handout 7 Practice asking open-ended questions. •

Apply the first three interview steps: 1. Put a person at ease 2. Explain your purpose 3. Ask the question

Explain which form(s) of corroboration you would use (source, receiver, object) Complete within 10 minutes •

§

Record Reviews •

Ask generators and users to point records out and describ e their use Best not to view records by yourself

•

•

§

Avoid asking for copies

§

§

Records are becoming virtual. Share some implications of this shift, and tips for managing. Practice respect. See if auditees know how to generate new records and access past records. Remember to include forms.

•

Tip

Copies can get lost and they generate uncertainty with the auditee. To help maintain document security, take good notes so you won’t need a copy. Review the difference between documents vs. records (documents come before production, records come after).

© ASQ 2012

Briefly look over the handout with the students, and explain this is a form for them to use on the job if they’d like. Provide guidance on the difference between openand closed-ended questions, and corroboration identification as needed. Invite learners to volunteer and encourage participation. Complete activity within 10 minutes.

70

§

§

Share some implications and tips for managing virtual records.


How to observe operations:

Review the difference between documents vs. records (documents come before production, records come after).

Tip

•

For Example

•

Practice respect, but above all safety. Get close to the machines and equipment. To help encourage detailed communication, ask the machine your related questions, not the person. Take notes of everything.

Check the machine settings against the procedure you read to prepare. Because you prepared, you know the difference is there. The auditee then tells the truth in detail. The operator responds to the machine, not the auditor. Shift focus from the auditor to the thing being audited.

© ASQ 2012

71

Audit Performance

Field Notes Examples §

Bank Example: Objective Facts •

•

•

•

•

•

•

•

•

•

•

•

•

© ASQ 2012

The computer network was down for a total of 25 minutes during the month of July at all teller stations. Tellers backed up data for the seven shift changes examined. The Hill Street branch experienced three cash machine paper receipt jams in June. Generic deposit slips were available at all teller stations. Vault lights are left on all night. Floors are wet-mopped only when customers are not present. Six of 48 overhead lamps were nonfunctional at the Charles Circle branch on Friday. All tellers are examined for math and communication skills prior to hire. Backup server was loaded with out-of-date data files on July 12. Cars using drive-in stations back up into the street. Bill-sorting machine malfunctioned on July 3 and again on July 9. Paper and plastic recycle bins are in all branch lobbies. ATM software runs on Win2000 and has never been patched.

§

§

The bold type in these examples emphasizes degree of detail. Suggest students use adjectives to cover the 5 W’s, How.

Check understanding of: 1. objective vs. subjective; 2. degree of detail; 3. conformance vs. nonconformance.

Continue to have learners identify conforming vs. nonconforming facts. Select a fact or two and reconstruct as a subjective, to emphasize the difference.

72

§

§

Invite learners to volunteer to: 1. Identify conforming vs. nonconforming facts 2. Identify specific details that are key to writing a good fact statement 3. Reconstruct a fact or two as a subjective statement

Use this example to continue to check for understanding of conformance/ nonconformance, objective/subjective, and degree of detail as needed.


Chemical Plant Example: Objective Facts •

•

•

•

•

•

•

•

•

•

•

•

•

•

13 out of 15 prototype problems were due to material issues with Smith Enterprises. Deficiency reports are used throughout the R&E department. Trend analysis is not performed in the R&E department. Shrinkwrap breakage restricts the wrapping line to half speed. Excel is shipping defective shrinkwrap. Ceramic adhesives shipments are all within spec since the line started last year. Process variance reports are all entered into the C/A system. Five out of nine unsat. analysis reports in the lab are due to high chlorine levels in samples. All five lab technicians are cross-trained, and their proficiency is checked annually. Sample bottles from Jones Lab Supply contaminate samples with halogens. Lab does not use the C/A system. All incoming material receives QC. 5/16 reject shipment reports from Smith Fabricators. 5/16 reject shipment reports from Jones Lab Supply.

© ASQ 2012

73

Audit Performance

§

Group Activity: Gather Data •

•

•

•

As a team, choose to explore the processes you “turtle-ized” Ask and answer interview questions One auditor and one auditee per interview o Use your checklist questions developed earlier o o Ask open-ended questions (remember: who, what, where, when, why, how, show me…) Provide both conforming and nonconforming answers o Four minutes per interview, then rotate around the table o After one round, convert your notes to statements Ask instructor for Round 2 directions

§

§

§

§

§

§

§

Let students know they are about to write their own data. Be available during group work to check for understanding and assist with questions. Before activity, ask students if they have any questions.

Choose a starting person in the group. Have team assign a rotating time-keeper to track fourminute interviews. Periodically alert students of time, and monitor progress. Prepare chart sheets. Have students convert their notes to statements and transfer them to the half-sheet in the appropriate column. For Round 2:

§

§

§

§

© ASQ 2012

Review statements and provide feedback. Guide learners through rewrites and monitor until each team writes at the target level. On the chart sheet, list a total of at least eight facts (some conforming, some nonconforming).

10-minute break

74


Module Review: Audit Performance Phase

Audit Performance Summary of Key Concepts In this module, you: •

•

•

•

•

•

•

•

•

•

•

Identified six key components to successful audit performance. Became familiar with action items and schedules for one- and two-day audits. Recognized how opening meetings can improve your relationship with auditees. Discussed meeting mechanics and topics. Prepared an opening meeting agenda and an audit agenda. Determined effective methods for communicating progress and issues to management. Tracing and its application to auditing The six steps for conducting an interview The importance of no secrets to all parties Common protocols for examining records and observing activities Objective evidence statements

© ASQ 2012

75

Audit Reporting

Audit Reporting

© ASQ 2012

76


Writing the Report Phase 3: Audit Reporting Instructor Section Objectives Ensure attendees meet all section learning objectives.

§


•



§


§

Chart sheet

§

Marker

•

•

Define audit finding and other terms Understand the steps to write a finding Identify root causes Recognize the differences between audit and inspection

In order to: •

•

Focus the organization on the system aspects of management Create finding sheet format and contents

Note: The Audit Report is your product. As an auditor, it’s what you make.

Instructor Demonstrates How to create finding sheet format and contents. §

Note

Audit Conclusions Instructor Note Orient learners to the part of the process represented in the general model for auditing. §

© ASQ 2012

77

Audit Reporting

Reporting Definitions

§

Finding: The ISO 19011 definition of a finding depends upon ISO9000. It is not typically used in the field. The above definition can apply to audits outside of quality (i.e., safety, environmental, sustainable, etc.). Conformity Assessment: Official term for registration certification. Conformity assessment mostly affects international trade in commodities. A third-party laboratory completes testing in an accredited environment agreed upon by buyer and supplier. Apply this concept to various ISO standards. All RAB-approved classes teach this kind of an audit under conformity assessment. Positive Practice: Remember from Day 1: it’s OK to say good things. Observation: Another term for “fact.” Gather them all to come up with findings, i.e., Findings of Special Examination Committee of Congress. The word first appeared in the 1980s as a mini-finding (want to write you up) as a precursor opportunity for improvement. The word is hardly used anymore; it makes sense intellectually but doesn’t work. Because observations are not requirements, no action is taken. The exception is with the Food and Drug Administration (FDA), who uses this term as it was originally meant. The FDA generates Form 483 (those in the field will know what this is) as a warning that if the recipient doesn’t fix what is noted, the FDA will stop production.

© ASQ 2012

Remember positive practice: it is OK to say good things from Day 1.

78


You must define any terms you use in the: •

•

Instructor Note Remind learners that defining terms helps improve clear communications and prevent misunderstandings that can lead to weakening of your relationship. §

§

Ask learners if they have audit procedures. If they don’t, prompt them to write this in the “A Box” of Handout 1.

Instructor Note Explain the data dump process. §

Audit procedures Actual report

Defining terms helps improve clear communications and prevent misunderstandings that can lead to weakening of your relationship. Auditors can use any terms you wish, as long as you define them. Does your organization have audit procedures? If not, write this in the “A Box” of Handout 1.

Create Audit Findings in Three Steps Step 1: Data Dump

The data dump process: Team members share the good facts and bad facts that were gathered. The team leader is the scribe that writes them all down. There must b e no sorting, evaluation, ranking, etc.—just get everything on one page. You must not get wrapped up in analyzing yet. Note: This is the same thing you practiced during the interview activity.

© ASQ 2012

79

Audit Reporting

Step 2: Data Chunk

Tip

When organizing data, look for similarities, common actions, common consequences, and relationships, then place them in related piles. If the issue relates to two different things, make a copy and put in both piles. Some data doesn’t appear to have a pattern.

§

§

© ASQ 2012

This revisits the Chemical Plant Example from Day 1. Remind students to focus on the big piles.

80


Step 2: Data Chunk: Chemical Plant Example

Piles won’t all be the same size. As auditors, focus on the big piles.

Note

© ASQ 2012

81

Audit Reporting

§

Step 2: Data Chunk: Bank Example §

This revisits the Bank Example from Day 1. Focus on the big piles.

Step 3: Create Findings Sheet Statement of the problem •

•

•

Bad fact Bad fact Bad fact

Now, turn your piles over. Each pile becomes a findings sheet. If the conclusions are reasonable, the facts will yield the same conclusions no matter who draws them.

Instructor Note Explain to students that finding at the top is always subjective opinion. Statements underneath are objective (facts). §

§

© ASQ 2012

Emphasize the difference between audit and inspection.

82


Step 3: Create Findings Sheet: Bank Example

The finding at the top is always a subjective opinion. The statements underneath are objective (facts). You may have seen facts presented as findings. Identifying corrective actions for facts only addresses the “symptoms” of the problem. Identifying corrective actions for the true finding allows you to “cure the disease.” Note

The fundamental difference between auditors and inspectors: inspectors tell us we have issues (they focus on the facts), while auditors tell us why we have problems (they focus on the findings).

© ASQ 2012

83

Audit Reporting

Write the Summary •

•

Overall evaluation of the management controls o One or two paragraphs o Very subjective (by design) Points to address Is the operation safe? Efficient? o Is there reason to believe that controls will continue? o o Are there any regulatory risks?

Cost production risk (CPR) drives everything. Each pile gets put on a findings sheet.

Note

© ASQ 2012

84

Auditing for Improvement Focus on the systems •

•

System: grouping of interrelated processes designed to achieve a common objective To have lasting effect (improvement), stakeholders must have a desire to change the system

Auditors are systems analyzers. Systems analysis gets the stakeholders to own these problems, not the auditors. Tip

This is the key to the report, because people are more likely to change because they want to, not because you tell them to change. (CPR)

Group Activity: Write a Finding Instructor Note Instruct the group to write a finding. §

§

Complete activity within 30 minutes.

1. Sort your data sheet by problem area 2. Assign a symbol to each problem area and corresponding data (some items may have more than one symbol) 3. Identify the most important problem 4. Write a finding for this problem on a new sheet of paper. Include: Important problem o Supporting objective evidence o

Complete within 30 minutes

© ASQ 2012

Audit Reporting

The

Audit Closing Meeting

The Closing Meeting Presenting the report •

Team leader conducts the meeting and gives the summary

Handouts to distribute •

•

•

Draft findings Positive practice sheets Example response sheet

Firms no longer make recommendations as part of the audit report for these reasons: •

•

•

•

Malicious compliance Inadequate knowledge Perceived bias (NIH) Ownership

Recommendations are obvious when findings are well written.

Obvious recommendation: Identify nonconformances.

© ASQ 2012

85

86


Audit Reports and Records The Audit Report §

Example of two-page report, plus three attachments. Background is the what, why, who, etc.

This is an example of a two-page audit report and three attachments. (Note: the background section is the who, what, why, etc.)

Group Activity: Write a Summary Instructor Note Instruct students to write a summary. §

§

Complete activity within 15 minutes.

1. On chart paper, write one to two paragraphs that address your audit conclusions. Be sure to include both conforming and nonconforming statements. 2. Answer the question: Are most management controls in place? 3. Post the summary close to your findings sheet.


© ASQ 2012

87

Audit Reporting

Compiling and Distributing Audit Reports Roles and responsibilities: The audit team works for the audit coordinator to write the report The audit team leader drafts cover letter to include:

Background Executive summary Request for corrective action The audit coordinator sends the report to the auditee •

•

•

The audit coordinator may not revise the audit report. Revisions must be made by the audit team, or else all credibility of the audit is lost. Alert Keep copies to a minimum in order to: Minimize confidentiality issues Promote trust and respect Note: Take precautions when distributing reports electronically to help maximize privacy and control. •

•

The audit coordinator is charged with providing senior management with a quarterly summary.

Note

© ASQ 2012

88


Audit Records

Instructor Note Remind students to transfer records requirements to the A or B box of Handout 1. §

Records requirements: In your audit procedure, define your long- and short-term records •

•

•

Which records? Where to keep them? How long to keep them? Keep yourself out of trouble by defining your records requirements in your local procedures.

Tip

© ASQ 2012

89

Audit Reporting

Module Review: Audit

Reporting Phase

Summary of Key Audit Report Concepts In this module, you learned about: •

•

•

•

•

Defining audit finding and related terms The need for corroboration and three common methods The steps to write a finding Identifying root causes The differences between audit and inspection

§

© ASQ 2012

10-minute break

90


© ASQ 2012

91

Audit Closure

Audit Closure

© ASQ 2012

92


Corrective Action

Output of the audit becomes input to corrective action.

§

Check in with students to determine if a break is necessary before you cover this section.

© ASQ 2012

93

Audit Closure

Compare and contrast these actions:

§

Corrective Action Request Form

Instructor Note It may be useful to walk through a findings sheet to illustrate. §

Finding requires corrective action

© ASQ 2012

Nonconformance (bullets) require remedial action

Review this example of a typical CAR form. The top part is the responsibility of the audit team. The middle part is called the “action plan” and completed by the auditee. Everyone around the planet requires response back in 30 days. The bottom part is the responsibility of the CAR Coordinator.

94


Remedial actions all start with R.

Note

Instructor Note Ask students to write in their manuals: fast, cheap, easy. This helps emphasize that as an auditor, you want to save energy. Don’t waste energy on the remedial action. Spend your time not on NCRs, but on the “disease.” §

§

Short-term actions •

•

•

•

•

Rework Reject No spec change ----------------------------------------------Repair Spec change Release Regrade o Recycle o

Remind students there are always four parts to corrective action.

© ASQ 2012

95

Audit Closure

Tip

The corrective action process is difficult to do, and where an auditor’s energy should be spent. Be careful! There is a tendency to put too many things into the corrective action system. Having threshold criteria will help manage this.

Group Activity: Respond to a CAR 1. Trade findings sheets 2. Prepare a response, including: Remedial action for each bullet o One root cause for problem o Three action items for problem (unlimited budg et!) o


Instructor Note Adjust break if necessary to break before this activity, or skip break and go straight through to the end. §

§

§

© ASQ 2012

Monitor the class work. Students may need assistance here. Optional 10-minute break.

96


Evaluate and Verify the Implementation Instructor Note Emphasize that CAR has to do with inspection, not effectiveness.

•

§

•

•

•

•

§

Response is your report card Nasty? You blew it! o o Agreement? Way to go! Look for system changes o We are in this for the long haul o Avoid the temptation to micro-manage Remember: change has cost o This is “inspection” only Did they meet the promises? o Effectiveness will show in future activities o Keep it simple Paper o o Test results or SPC data Monitoring by other staff o Close out By CAR coordinator (on form, with email notification o back to responsible parties)

Invite students to ask questions.

© ASQ 2012

Auditing for Improvement Course Summary

97


© ASQ 2012

98


Course Review

Phases of the Audit Instructor Note Briefly summarize the highlights of each phase. §

Effort Required by Audit Phase 10% Phase 4: Closure 25% Phase 1: Preparation

15% Phase 3: Reporting

50% Phase 2: Performance

Value of the Audit Audits provide confidence: •

•

•

•

Controls are present Controls are used Controls really work Controls will continue

Instructor Note This piece reminds students why they are doing audits. §

© ASQ 2012


99

Rules for the Auditor and the Audit You must: 1. 2. 3. 4.

Prepare Dig for threads and patterns Look for cause and effect Ensure stakeholders understand the language of business (cost, production, risk)

Instructor Note Emphasize these key success factors: cost, production, risk. §

© ASQ 2012

Student Version

Recommend Documents