Six Sigma Final Sample Project

Aveta Six Sigma Online Black Belt Course

SAMPLE Six Sigma Black Belt Project Report

1


Executive Summary

As a new subsidiary of an established company moving into a new business, establishing a reputation for solid customer service service at a low price is paramount for the Security Security Department’s new serivce. Along the way, the business manager discovered that the idea of a solid customer service reputation r eputation was in jeopardy, but felt the personal service service could not be beat. Under that presumption, presumption, he solicited help to use Six Sigma processes to 1) validate the the perception, 2) propose a recommendation recommendation for solutions and 3) 3) put processes in place to ensure control in the the long run. The Problem After research and survey, we found that site managers were pleased to be moving to the new video surveillance technology, but they were disappointed in the number of outages they were seeing in the equipment. After measuring the process, we found found there were several reasons for outages: 1) 2) 3) 4)

Network service issues issues Software update issues User training issues Equipment failure

Of these, the latter proved to be the most costly and unexplainable. According to information from the manufacturer, we should not see see the kinds of problems we were were having. The estimated cost for these repairs exceeds $70,000 annually and has an untold impact on Operations. The Process Using the Six Sigma process of Define-Measure-Analyze-Improve Define-Measure-Analyze-Improve-Control, -Control, we determined what the priority is for the Security technical technical group. We then chartered a project and proceeded proceeded to understand and define the problem. Through customer surveys, site visits visits and data gathering began to focus on the real issues at hand. To help clarify the situation, we mapped the installation and support support processes for the security equipment finding and correcting issues as we went. Eventually, the measurement and analysis processes pointed to the fact that we had a real issue that was out of the ordinary for the equipment equipment in service. Fortunately, we had clear concise records records and good support from the manufacturer to help research and resolve the problem. Findings As the old adage goes from a well-known Houston media personality, we had “slime in the ice machine!” More importantly we had significant amounts of dirt in the machines. A by-product of the process/project was that we discovered the Information Technology team was having the same issue, just not quite to the same extreme. With some research, the team did find a low cost solution to protect the machines. While it is early in the post-solution phase, phase, we are seeing improvements. Conclusion By using a systematic approach, approach, we identified several areas in the the process that needed modification. modification. The net result is a longer machine life with an expected savings of more than $70,000 annually as well as improved customer service. Based on the proactive response response to their issues, the Operations Operations Management teams of both the internal and external customers are beyond satisfied and are serving as excellent references for the Security Department.

2


Definition Phase

Project Opportunities As a new subsidiary business within a well established, large company, the Security Department has several competing projects for a small staff. They find themselves needing to balance ongoing operations, roll out of more services to the parent company and new sales opportunities for external customers. For internal customers, the goal is to provide excellent security service burglar, fire and equipment monitoring - as well as video coverage (with historical recordings) of the operations area. For external customers, their strategy is to utilize their existing staff to provide low cost services for monitoring alarms. Their goal continues to be low price and excellent customer service. Excellent customer service is defined as reducing false alarms and properly maintaining equipment for legitimate alarms as well as providing video services for those wanting to pay for it. In order to fully understand what was important to the customers, we conducted a survey to understand what people are expecting from the Security Department. From that survey, table 1 lists potential projects1 identified from on-going operations and for future customer sales. Weight

Add new model DVR support Resolve perceived DVR failure issue Automatic video retrieval on alarm Implement IP alarm monitoring function

Criteria and weights 4 2 3 Profitability Easy to Low Do Cost 3 6 8 8 2 3 6 3 3 6 4 5

10 Customer Satisfaction 7 10 6 8

Total 118 145 99 127

Table 1: Security Project List & Impact

In discussions with senior management, internal customers, call center employees and potential external customers, the Security Department used a weighting to help determine what to focus on first. Based on input, they weighed customer satisfaction as critical since their predominant advertising mechanism is going to be word of mouth for the first year of the new subsidiary. By using the results from Table 1, the Security Department decided to investigate the perception of DVR failures since it had a direct link to customer services. As a side note, this was not even on their list of potential projects prior to the survey. Voice of the Customer With the relative surprise response of a high rate of Digital Video Recorder (DVR) failure problems, I felt it was necessary for the department management (potential project sponsor) to really understand the nature of what the customer had to say. So rather than just presenting statistics and summaries from surveys, we discussed the comment section from each response. Site Manager (parent company) - “Being able to review and monitor transaction processing via video is critical. I never would have thought I would say that since I avoided the technology. Now, I can’t live without it. When the camera systems are down, it costs me money. If everyone knows it is down, I can’t image what they are doing.” 1

Actual list is longer. The list was truncated to simplify this report

3


Safety Manager (parent company) - “By having video access (and more importantly recording), I have ‘eyes in the back of my head’. This technology extends my ability to cover a site and find issues before they have a negative impact on the business. An outage, especially an extended one, raises a great deal of concern.” Security Director (external customer, multiple sites) – “The potential of the service is incredible, but the number of camera failures is frustrating. If we do not resolve this problem, then I will have to cut services back to the basics.” While these represent the extreme comments, several people made similar comments. The last one (and others like it) is what made Management sit up and take notice of a problem. Based on the scoring, conversations regarding customer service and potential impacts to their reputation of service challenges, the Company decided to authorize a project to investigate the perception and implement solutions if the allegations proved to be true. Remember this is a new subsidiary and working under “word of mouth” advertising. Project Charter A summary (retyped version to simplify project submission) is shown in Table 2 below. The original has full company and employee names along with telephone numbers and signatures.

PROJECT INFORMATION Project Name: Investigate & Resolve DVR Failure

Chartered Date: 6/1/06

Sponsoring Organization: Security Department

Project Start Date: 6/2/06

Revision Number: 2

Target Completion: 6/30/06

TEAM SPONSORS Project Sponsor: Doug – Security Monitoring Director

Contact Number:

Project Black Belt Tom

Contact Number

Project Green Belt None available

Contact Number

ADDITIONAL TEAM MEMBERS Name Role Matt Alex Sharron

Contact Number

Security Supervisor Technician Customer Service Rep

4


ADDITIONAL TEAM MEMBERS (continued) Barry Jeff Nick

Installation Specialist IT Analyst DVR Vendor Representative

PRINCIPLE STAKEHOLDERS Name Role Zack Bill Doug SLT

VP – Security ABC Customer – Security Manager Security Monitoring Director & Sponsor Senior Leadership Team

PROJECT GOALS Identify any deficiency in DVR performance. Assuming a deficiency, implement corrective actions to resolve the problem and return performance to documented performance standards. In the absence of a deficiency, document performance expectations and discuss with the customers. The following was added in revision 2 of the project charter after finding installation issues: Correct the installation problems and return DVR performance to p ublished performance standards as well as reinstate customer confidence in both the equipment and customer service. PROCESS PROBLEM The following was added in revision 2 of the project charter after finding installation issues: Under certain installation conditions, the DVRs are failing prematurely. The apparent cause of the issue is dirt. Certain operational environments are outside recommended standards – the actual offices are temporary spaces that have a high volume of traffic in a construction type environment. The result is the server fans are pulling in too much dust and dirt for the servers to function properly. SCOPE OF PROJECT The following are the items that have been identified as the scope requirements: 1) Conduct further surveys to understand the nature of DVR failures 2) Investigate performance standards and expectations of the various DVR brands 3) Compare installed DVRs and maintenance issues against the performance standards. 4) If performance is found to be substandard, investigate what is causing the problem and recommend solutions. 5) Pending findings and approval for #4, implement recommended solutions.

5


PROCESS IMPORTANCE The video support function is a significant value added service provided by the Security department. Some of the potential savings and findings are listed below: Monitor cash point of sales to discover fraud. One site reported savings of $500-$750 per day. The reports indicated that it had been going on for years. While that kind of fraud is extreme and rare, it does occur. Allows Internal Audit in performing site audits on viewing scale transactions with point of sale entries. Each IA trip is estimated to cost $5000. Using reliable video, IA could save at least 10 trips annually. In addition to these findings, the external customers are expecting to save a great deal on safety audits and inventory shrinkage discovery. While they have not documented their savings to the company, it is clear they are expecting to save over $100,000 annually. As the project team interviews the stakeholders to confirm their interest in the project, they will document more regarding cost savings. AUTHORIZED RESOURCES Project sponsor – Doug, Security Monitoring Director. As project sponsor, Doug is responsible for the following: 1) ensuring staff members are available to work as a team member, 2) coordinating funding with the management team using the figures provided by the project manager and black belt candidate, 3) coordinating participation from the customer representative. Doug’s role also has him serving in a sales role, so he maintains the customer relationship. Key Stakeholder – Bill, ABC Customer Security Manager. As customer, Bill is a key stakeholder in buying the service. He has a vested interest in seeing the project work as planned due to the amount of hard dollar savings. Additionally, there are some un-quantified savings that he is expecting to achieve. His role on the project is to provide as much detail as possible about the operating environment of his company as well as answer questions regarding expectations and understanding service deliveries.

Key Stakeholder – Zack, VP – Security. In addition to being a key stakeholder, Zack is also responsible for the entire Secure Department budget (he has fiscal and functional responsibility for the Security Department). As a stakeholder, his interest is to ensure that customer service is maintained. He services as one of the key people in regard to sales and is a major proponent of the group. While the sponsor will facilitate getting a funding stream, Zack will also help. He is also responsible for answering to the Senior Leadership Team 6


AUTHORIZED RESOURCES (continued): in regard to external sales. As a result, his role will be to help drive an understanding within the leadership team of 1) problem resolution techniques and 2) capital funding for implementing the project resolution. Key Stakeholder – Senior Leadership Team (6 people) The senior leadership team represents the stockholders and outside interest of the company. They have authorized and provided funding for the new subsidiary. As such, they want to receive a return on investment (profit) for that funding; however, they owe support to the operation to ensure that the best quality of service can be delivered. In the event that it cannot produce the profit as expected, they will own the decision to either continue or cease operations. Team Members – Matt Alex Sharron Barry Jeff Nick

Security Supervisor Technician Customer Service Representative Installation Specialist IT Analyst DVR Vendor Representative

Duties are to be defined as the project proceeds. Outside Needs Some travel to locations may be required as part of understanding the installation process. The point of having experienced team members (Jeff, Nick and Barry) on the team is to minimize that need since they have worked in the various sites. Technical expertise regarding the equipment in service will be required. As a representative of the company providing the equipment, Nick has access to that information. A test system with cameras may be needed for experimentation and configuration changes if required. Training needs are unknown at this time, but expected to be minimal based on the experience of those selected to participate. They were chosen under the theory that they have “seen everything”. Table 2: Project Charter

7

Aveta Six Sigma Online Black Belt Course Project Schedule Figure 1 is a schedule drafted early in the discovery process the edited as data became available. The control phase is documented in later sections. ID

Task Name

1

Definition Phase

Duration 23.32 days

2

High level business need survey

3

Review findings

4

Meet with business leaders regarding findings

5

Detailed follow-up on customer issue regarding DVRs

6

Follow-up meeting with business leaders regarding problem

0.33 days

7

Identify customers, sponsors and process owners

0.33 days

8

Identify resource requirements - high level

0.34 days

9

5 days 1 day 0.33 days

Team selection

3 days

0.33 days

10

Draft project schedule

2 days

11

Draft project budget

12

Review with sponsor

0.33 days

13

Project Charter approval

0.33 days

14

Detail project plan

2 days

15

Process Mapping

4 days

1 day

16

Develop maps

1 day

17

Resolve immediate issues (clear gaps in process)

3 days

18

Develop success metrics (CTQs)

1 day

19

Evaluate existing data for applicability

1 day

20

Sponsor review

1 day

21

Measure Phase

5.32 days

22

Validate metrics

23

Review existing data

24

Develop checklist measurement & sample sizes

25

Prep base analysis charts

1 day

26

Re-evaluate source data.

2 days

27

0.33 days 1 day 0.33 days

An alyze Ph ase

3.32 days

28

Evaluate preliminary chart results

0.33 days

29

Benchmark to published standards

0.33 days

30

Compare results across location type

0.33 days

31

Compare results with vendor

0.33 days

32

Review results with business leaders

0.33 days

33 34

Improve Phase

28.33 days

Team brainstorming sessions

0.5 days

35

Options analysis (costs & time)

36

Review options with business leaders

37

Capital approval documentation

2 days

38

Pilot on 2 sites

5 days

39

Rollout to remaining sites

40 41

0.5 days 0.33 days

20 days

Control Phase

2.16 days

Determine control variables

0.33 days

Figure 1: Project Schedule

8

Week 1

Aveta Six Sigma Online Black Belt Course Process map Since the results of the original survey were a surprise to the security team, we documented the setup process (figure 2) as well as the trouble shooting and repair processes (figure 3) as well.

DVR Installation Process Request review by Security Dept (cost & equipment estimate)

Request from Site for Video

Cost review by site management & Security

Implement Video?

Yes

Security orders equipment

No

Counterproposal for other options

Equipment staging (separate process chart for this) Yes

Cost Issue?

No

Site testing

Site configuration

Local IT or contractor installs equipment

No implementation

Site prepares for data and power requirements

Security arranges for local installation

Ship equipment to site

No

Remote testing

Yes

Test OK?

Production Ready

Figure 2: Setup Process

Special notes: As a by product of creating the process maps for system setup, we discovered three significant points. First, some items were not being configured consistently or at all. Some of that depended on the technician handling the process. The second item is that the sites were not necessarily prepared for the implementation process. In many cases, the local installation technician needed to arrange for power and data cabling once on site. That loop is shown in the setup process above. The third was the need to create a knowledge base for troubleshooting (figure 3 – troubleshooting). In evaluating the process, one technician was generally perceived as a bottleneck. In reality, he had a better process for handling issues which resulted in a better resolution rate. That is not documented in the results for this particular project.

9


DVR Troubleshooting & Repair Process Automatic reporting function sends trouble signal

Technician calls site to discuss problem

Return to production No

Problem exists?

Onsite personel notice issue and call in

Dispatch regional camera contractor

Yes

Technician works troubleshooting process*

Full system outage?

A Yes

User resolvable with instruction?

No

Review camera specific issue

Camera contract contacts central technician to diagnose & resolve

Review DVR processes with user (training as required)

Issue resolved

Yes

Reboot DVR No

No

Yes

No

No

Camera specific issue

Yes

DVR Functional Issue?

Restart normal?

No Yes

A

No

Issue resolved

Yes

Return to production

Yes

Yes

Resolve per Microsoft or DVR vendor specifications & standards

Issue resolved

No

Hardware repaired and returned per set up process

Review error messages

Customer ship unit to central support

Yes

Software Error?

No

Hardware error?

* Common issues are camera outage, focus issue, bad lighting (camera contrast), can not search video (training issue).

Figure 3: Troubleshooting & Repair Process (Simplified)

Key Measures for Evaluation From the troubleshooting and repair process, we found that several machines were being returned to the central support group. The general perception is that anytime a machine is taken out to be shipped, the overall outage is entirely too long and expensive. From that information and perception, we were in a position to evaluate run times, environment and impact to the business. Some of the items that we felt are critical to customer satisfaction or needed to be understood for comparison purposes are as follows (the list grew after the first analysis iteration):

10

Aveta Six Sigma Online Black Belt Course 1) Types of outages for the DVRs. a. Software issues b. Hardware issues c. Network outages (provider service) d. Network issue (hardware other than DVR, non-provider) e. User error f. Miscellaneous 2) Vendor performance expectations a. Warranty and service life for the unit b. Operating conditions 3) Number of units returned to central support in various time frames a. Weekly b. Monthly 4) Total number of support calls resulting in on-site technical support a. Weekly b. Monthly 5) Total duration of outages over various time frames a. Weekly b. Monthly 6) Total number of units in production In addition, we did discover that the central technical support staff was keeping detailed notes on the various things they were finding with each returned machine. They were just not making a point of discussing how many machines they were seeing with the management team. In the long run, the notes saved a great deal of time in getting to the final solution. That is noted in the section discussing the Analyze Phase to follow. Lastly, we also wanted to make sure that we understood the impact to the customer for each outage. The general opinion was certain outages were “less expensive” than others. We would use the Measurement Phase to help drive where to look: 1) Hard dollar cost per repair 2) Soft dollar cost per hour outage 3) Total outage per site over various periods a. Weekly b. Monthly

Measure & Analyze Phases

Using the information from above regarding the key measures, we then determined that the critical quality variable is system availability. That is generally stated as operating hours per month at the DVR level or more importantly as the time in service before failure. In some cases, the camera level data also becomes important, but not all cameras are equivalent. In order to keep the measurement process at a definable level, we focused at the DVR level overall. Camera level analysis at specific locations may become a target for review in the future. Following the decision to treat issues at the DVR level, we worked to understand the relationship among the different items as they relate to cost and business impact. The following are some of the items that we discovered:

11


1) The hard dollar costs are a dependent item, so tracking them independently is not required. 2) Network outages generally do not have a long term negative impact. The DVR continues to record and the data can be accessed once the network is available. As a result, we only used 4 weeks worth of data for analysis purposes. A later study will work with the external network provider to improve that service. 3) Opinions on soft dollar outages vary significantly based on location type. For example, sites with high volume of cash transactions (Operations) feel an outage is significantly more expensive than an office environment. As such, those numbers are not included in outage estimates in order to be more conservative on savings. Any presentation to business leaders did discuss potential numbers around soft costs. The end result is that the key variables for monitoring are the following: 1) Number of hours of outage – relates to total cost for the outage 2) Number of units returned to central support 3) Operating time to failure The latter uses the definition of failure as some event that requires removing the machine for maintenance (actual failure or diagnostics). Appendix A lists some of the raw data used to arrive at the conclusions listed above and to create the Pareto chart shown in Figure 4 & 5. Due to the correlation between labor effort and outage costs, Figure 5 pointed to the area needing attention – specifically that the DVR hardware maintenance uses more labor. The next question needing research was to find out how the DVRs were performing in relation to performance specifications. Some of the data for figure 5 as well as subsequent analysis came directly from the central support maintenance logs. A sample log sheet is shown in Appendix B. Total Outage Counts by Type

160

Count of Cause of Outage

140

120

100

80

60

40

20

0 Network

DVR Hardware

User issue Cause of Outage

Figure 4: Count of Outages by Type Network outages for 4 weeks only. All other data for sample time frame

12

Software

Network Hardware


Total Technician Hours by Cause of Outage

350

Sum of Technician Hours

300

250

200 Total 150

100

50

0 DVR Hardware

Network Hardware

Software

User issue

Network

Cause of Outage

Figure 5: Total Technician Hours by Cause

Taking figure 5 a step further, we then analyzed the outages over the sample time. Figure 6 shows the findings. Using the outages per week against an upper limit (lower limit was set to zero since it could not be negative), we found the actual outages to be “in control” meaning that whatever perception site management may have, it was consistent and unless we changed something, it would continue; however, the perception remains that there were too many outages for their expectations. Outages per week 6 5 s e g 4 a t u O f 3 o t n u 2 o C

1 0 1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

Week Outages

Average

Figure 6: Outages per week

13

UL


Taking this information and translating that into business impact, each outage has an average cost of $597 in repairs (labor, parts, shipping), an average outage of 70.15 hours and 2.44 outages per week. (Appendices B and C show cost and outage information.) The distribution of the outages is shown in Figure 6 previously. This does not attempt to quantify the “soft” dollar costs since the various locations have such a diverse definition of what the cameras are saving them. However, we do know that Internal Audit has a concern about the availability of the video to the point that they have not cancelled any trip in lieu of video coverage. In order for Internal Audit to “get comfortable” with video coverage and to ease site managements concern about outages, the management team mandated cuting the outages from an average of over 2 per week to 2 per month (1 every other week). That mandate equated to more than $60,000 savings annually in hard dollars. Again, the soft dollar costs are not documented. The questions needing to be addressed where 1) is this possible within the environment and 2) what would it take? The first question addresses the issue around why are the DVRs failing and is the failure within the manufacturing tolerances. The second question is based on the assumption that there was something within the environment causing the problem and not a manufacturing issue. In order to understand how the failure rate compared to what should be expected, we went back to the original installation information. While much of the information was available, there were some assumptions needed: 1) DVRs would be in production 24 hours per day 2) DVRs would be in production every day from installation 3) Production is defined as turned on so that the hardware components were in use even if not for the intended software package. 4) The exact installation dates are not available, so we used the first day of the month for the most conservative estimates of time in production. Using those assumptions, we then compared observed service life with expected performance. While expected performance can be expressed in many terms, we elected to use a conservative definition of warranty period.2 Based on that expectation, we found the following: Number of Failures Total Population Average Time to Failure (days) Warranty Period

39 508 382.87 1080

Maintenance records were sampled for a 16 week period from 1/29 to 5/19. Warranty is 3 years @ 360 days/year.

To further understand the outages and impact, we then analyzed which sites were experiencing the problems. That also relates to the actual business impact of the outages. In theory, zero revenue sites have less at stake (soft cost) than revenue sites. Table 3 shows the largest portion of outages effect operational facilities. Table 4 gives the installation base as background.

14

Aveta Six Sigma Online Black Belt Course Count of Cause of Outage Location Type Cause of Outage Not documented User issue Not documented Total Office DVR Hardware Network Hardware Software Office Total Operations DVR Hardware Software Operations Total Transfer DVR Hardware Network Hardware Software Transfer Total Grand Total

Total 39 39 2 1 2 5 34 2 36 3 1 1 5 85

Table 3: Outage counts by location type and cause

Count of DVR Number Location Type Total % of Total Office 26 5% Operations 443 87% Transfer 39 8% Grand Total 508 Table 4: DVR counts by location type

We also took one more assumption – namely that all DVRs that have not failed will not fail before reaching the warranty level (3 years). With that assumption, we then recalculated the average time to failure as 1026.48 days with a standard deviation of 187.84 days. When plotting that in a control chart, most all of the failed units are outside of the control limits. Figures 7 and 8 show that information (the tools are limited so that 508 points cannot be included in one chart). These assumptions putting the failures outside of the control limits are conservative. Most notably is that our sample mean is less than 3 years. The actual mean from the manufacturer is more than three years – thus the warranty set at that point. A call to the manufacturer outlining our problem got us this information. That call and subsequent discussions resulted in finding that they are not seeing out of the ordinary challenges like this. After that call, they have cooperated in our investigation by sending a systems engineer to work with us through the later phases of the project.

15


DVR Produc tion Days 1800 1600 1400 n o 1200 i t c u d 1000 o r P n 800 i s y a 600 D

400 200 0 DVR (1-255)

Figure 7: Production Days by DVR (1-255) DVR Production Days 1800 1600 1400 n o 1200 i t c u d 1000 o r P n 800 i s y a 600 D

400 200 0 DVR (256-508)

Figure 8: Production Days by DVR (256-508)

DVR Production Days Against Manufacturer Information 2000 1800 1600 n 1400 o i t c 1200 u d o r 1000 P n i s y a D

800 600 400 200

0 Failed DVRs Time In Production

Figure 9: DVR Failures Against Manufacturer's Information

Based on this new data (or more accurately, based on this relevant comparison), we can conclude that we are not seeing the expected life we should see. Something is causing the DVRs to fail 16


prematurely. At that point, the team used a brain storming session to develop the cause and effect diagram shown in figure 7. Many of the items shown on the diagram were directly from the technicians’ maintenance notes (sample form shown in Appendix B). While the manufacturing area is a process that could have problems, we tended to focus on the items that were handled internally. Everyone is aware that some machines will fail prematurely due to manufacturing challenges. If we get to the point where that is the only cause, then we will initiate another project to resolve that problem. Manufacturing

Installation

Damage in shipping

Improper wiring

DVR Failure Power surges Out of date drivers

Dirt

Missing virus software

No Preventive Maintenance Untrained users

Configuration & Staging

Production Environment

Figure 10: DVR Failure Cause & Effect

Improvement Phase

Using the information from Figure 10 and the maintenance logs, we took each of the causes and evaluated their true impact on the problem: 1) Out of date drivers and missing virus software While these are a problem that this discovery process did find (and have since corrected), this would not cause the kind of outages we were seeing. A follow-up virus scan of the remaining DVRs in production turned up no infestations. 2) Damage in shipping This could be an issue, but we use the shipping containers from the manufacturer to send the machines to the location. The systems engineer confirmed that the manufacture is not seeing out of the ordinary challenges. 3) Improper wiring & power surges This does explain some of the outages we have seen. The maintenance notes on one of the power supply outages clearly documented a surge. It was not a standard policy to have surge protectors. That has changed. 4) No preventive maintenance After analyzing this, the team concluded that this may have saved the faulted units by finding the other conditions before they caused an outage. 17


5) Untrained users This is an issue for the technicians handling telephone calls. It is not a cause of hardware failure. The user security and software prevent the users from damaging the data. This topic will be analyzed for a future project. 6) Dirt After reviewing the maintenance notes, this jumped out as the most likely cause of issues in the operating locations. All of the maintenance logs for operations locations referenced an abnormal amount of dirt in the machines. Based on that, the installation specialist, technician and systems engineer from the vendor we nt to a field location – specifically to a site of a failed DVR. The following outlines their statements: Technician – “No wonder the machine failed, this place is a mess” Systems engineer - “We test machines in extreme environments, but this is beyond what we would expect to see normally.” Installation specialist after hearing their comments – “This is normal for operations” The operating locations tend to be temporary buildings much like you would find on a construction site. One of the external customers is actually a construction firm. Generally, there is a lot of dirt movement with large equipment causing a dusty environment all around. The other observation was that the machines were not being carefully placed. Basically, the site personnel tend to direct where the machines will be placed and thinking of the machine’s health is not high on their list. Based on the maintenance notes and site observations, we needed to find a way to protect the equipment. Once again, the team headed into a brain storming session. Unfortunately, this time did not produce a lot of results; however, Google™ saved the day. There are several inexpensive covers and keyboard skins on the market. The resulting action plan followed: 1) All DVRs (and servers after a discussion with the IT group) in operations will have dust covers. 2) All keyboards will have keyboard skins 3) IT staff will perform routine cleaning of the DVRs and servers when on site. Compressed air cans are now standard issue. Further checks will be needed to see if the above recommendations are necessary at transfer stations and office locations. To pilot the process, we pulled two machines at the same site (some sites have multiple DVRs) from service for a quick preventive maintenance routine and cleaned them thoroughly. As they went back into service, one was covered with a dust cover and key board skin and the other was not. After a week, the team went back to the site to review the state of the machines. While it is not conclusive, there was a noticeable difference in the machines. Only a longer period of time and close review of the control charts of outages will clarify the understanding.

18


Control Phase As noted throughout this document, there were several places where we discovered minor deviations from best practices. As part of the ongoing control, there are several steps that are in place (or soon will be) to control and monitor the production environment: 1) A configuration check list has been standardized 2) Each technician completes the check list and signs it. 3) Someone other than the technician reviews the checklist for completeness and prepares the machines for shipping. The same person did step 2 and 3 previously. 4) Field IT will now handle preventive maintenance when at the sites working normal support. They will note any issues and report to the Security technicians. 5) Control charts like the one in Figure 6 are now posted for the top 4 items creating DVR outages. Any time complete failures happen once per week, the staff reviews the machine to look for issues. 6) Routine comparisons of equipment trouble tickets will be compared to what the IT group is experiencing to watch for similarities. As a side note, the IT group has implemented several of the things the Security department has found and while they have not published expected savings, it is easily going to be more the $100,000 annually when combined with the Security Department. As you can tell from the brevity of the control section, we have not had time to confirm the results. This has literally just been completed. Early indications from Security and IT are things are definitely on the right track. Continued observation will tell the tale.

Conclusion The net result of the project was more than anticipated in hard dollar savings. While we have not proven that we are down to one outage every other week, the early indications are good. In addition, the IT group is also seeing a reduced hardware outage issue, but not quite to the same degree. They had a better preventive maintenance program in place, but protecting the machines and discussing the issue with site management has proven productive. At this point, the estimated savings from this project is in excess of $70,000 hard dollars. As stated many times above, there is no estimate on soft dollars. In addition to that, we have also implemented new policies and practices. Some of those lessons learned include the following: 1) 2) 3) 4)

Virus protection & device drivers updates and maintenance Proper site preparations (wiring) Surge protection Servers were having similar issues

By using a systematic approach and truly analyzing the problem, we improved customer service, proved to be responsive and saved money.

19


APPENDIX A – Cost Structures & Support Hour Samples The following are the cost and implementation elements used for estimating outage impacts:

Function Contract labor – on site camera work DVR Shipping Central Technical support Network connection (Frame) DVRs in production Sites with DVRs

Cost $75/hour $50 each way $20/hour $1000/month/site 508 432

The following are the labor estimates for configuring, installing and troubleshooting (many of these are subject to future study for improvement via Six Sigma): Function DVR setup (from un-boxing to repackaging Telephone Support Software download & configuration On-site setup & configuration of DVR

Responsible Party Central Technical support

Labor Estimate 2 hours

Central Technical support Central support Contract Labor

15 minutes/call 30 minutes each 2 hours

20


APPENDIX B – DVR Maintenance Log Sheet The following is a blank maintenance log sheet as generated by the maintenance ticket system. The process has the technician manually complete the form. Eventually, the data is entered into the tracking system. MAINTENANCE LOG SHEET Date Received: Repair Ticket Number: auto-generated DVR Location: Configuration: Manufacturer: ____________________ Model: _____________________ Disk Drive type/size: ________________ Operating System: __________________ Address used for return shipment (Y/N): Equipment Status on Receipt:

Repair Comments:

Software Load:

Date Returned: Shipment Tracking Number: Technician:

21


APPENDIX C – Data Summaries Summary of outages by weeks of the sample: Sum of Outage Duration in hours Week of Outage 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Grand Total

Total 186 281 106 200 217 145 169 265 290 170 73 204 97 241 97 145 2883

Summary of Outage Hours by outage type Sum of Outage Duration in hours Cause of Outage DVR Hardware Network Network Hardware Software User issue Grand Total

Total 2736 77 36 25 10 2883

Summary of Technician Hours by outage type Sum of Technician Hours Cause of Outage DVR Hardware Network Hardware Software User issue Network Grand Total

Total 312 36 25 9.75 0 382.75

22


Count of Outages by Type Count of Cause of Outage Cause of Outage Network DVR Hardware User issue Software Network Hardware Grand Total

Total 142 39 39 5 2 227

DVR Failure counts by week and location type Count of Cause of Outage Week of Outage Location Type 1 Operations 2 Operations Transfer 3 Operations 4 Operations 5 Operations Transfer 6 Operations 7 Operations 8 Office Operations 9 Operations 10 Operations 11 Operations 12 Operations Transfer 13 Operations 14 Office Operations 15 Operations 16 Operations

Cause of Outage Total DVR Hardware 2 DVR Hardware 3 DVR Hardware 1 DVR Hardware 1 DVR Hardware 2 DVR Hardware 2 DVR Hardware 1 DVR Hardware 2 DVR Hardware 3 DVR Hardware 1 DVR Hardware 2 DVR Hardware 4 DVR Hardware 2 DVR Hardware 1 DVR Hardware 2 DVR Hardware 1 DVR Hardware 2 DVR Hardware 1 DVR Hardware 3 DVR Hardware 1 DVR Hardware 2

23

Six Sigma Final Sample Project

Recommend Documents