Build your knowledge base Rapid Problem Resolution (RPR) explained
Introduction 87% of IT problems reported to your Service Desk get fixed within hours, if not minutes. A further 11% are resolved resolved nd rd within days by 2 and 3 line support, perhaps with help from supplier technical staff. The final 2% are the toughest and represent chronic problems that are fixed by “trial & error”, or simply remain unresolved. Unfortunately, “trial & error” is very slow, sometimes expensive and often disruptive.
The good news is that we are at the dawn of a new era in problem resolution. The wide adoption of ITIL has provided a framework for the management of incidents and problems, and this in turn is driving an interest in problem resolution methods, particularly those based on Root Cause Analysis (RCA). One such method is A dvance7‟s Rapid Problem Resolution (RPR) Technique. In this paper we look at the need for RPR, when we should use it, how it works, and the skills we need to practice it, and its challenges & limitations. We finish the paper with a very recent case study that helps demonstrate the benefits.
The Need for RPR This question actually comes in two parts; why do we need a problem resolution method, and, why RPR? A problem reported to the Service Desk will go through four phases: Phase 1 - Fixed by Help Desk / 1st Line staff - Simple problems and user errors - Fixed through knowledge and procedures - Fixed within 16 hours worked Phase 2 - Dealt with by 2nd / 3rd Line Support - Issues caused by faults, overload or misconfiguration. - Fixed through advanced knowledge, tools and knowledge-base access - Fixed within a further 24 hours worked Phase 3 - Dealt with by 3rd line support with supplier product specialist - Complex problems, often performance related and/or intermittent 1 - Fixed through pattern method , detailed product knowledge and advanced tools - May use process of elimination - Fixed within a further 24 working hours Phase 4 - More and more people get involved - Complex problems that have dropped through Phase 3 process with root cause unknown - Attempts are made to fix through swap-out, holistic method, gut feel, random upgrades, etc. - May be fixed within 10 days to 2+ years - May never be fixed.
We can put this in an ITIL context. Incident Management covers all of Phase 1 and much of Phase 2 in that most of these issues are dealt with through any action that achieves rapid service recovery. Where the IT Team identifies
1
Pattern Method is at the core of a number of problem resolution methods and involves linking the first indication of a problem, a change in frequency or the pattern of problems to a common cause e.g. only Windows XP users experience the problem, therefore, the problem must be related to Windows XP.
Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
repeated occurrences of the same problem the issue moves into Problem Management. Such problems are typical of those that drop through Phase 2, often are fixed in Phase 3 and sometimes end up in Phase 4.
In Phase 4, the combination of procedures, skills and tools available in a typical company has been unable to determine the cause of the problem. At this point the IT Team is faced with two choices: 1
Do more of the same, even though this has not produced the solution to date
2
Resort to “trial and error” – which could involve testing a range of changes, upgrading infrastructure, or replacing software
There is a third way – method-based problem resolution, and there are many methods available. Some have sprung from the IT industry, but many of the front-runners are actually adaptations of business problem resolution methods. There are some common shortcomings: Due to the “soft” nature of many business problems, methods with this lineage are not designed to take advantage of the logic and tools available to us in the IT industry Many of the “methods” are actually just processes with no supporting IT techniques making it difficult for IT people to run the process
Many methods require that we already know the root cause as one of a list of possible causes that are then tested To avoid disruption, some methods force the IT Team to attempt to recreate the problem in a lab environment, which is time consuming, can be expensive and rarely works Many methods rely on statistical analysis which often fails with Phase 4 problems due to their intermittent nature and transient causes Some methods rely on trial and error Although many methods claim to be based on root cause analysis most only achieve this with hindsight
RPR was designed from the outset to solve IT problems and is heavily influenced by software engineering techniques, 2 primarily IBM‟s PSI/PD . From this starting point RPR avoids the shortcomings suffered by other methods because: It makes full use of the IT tools that are available in every business It is a fully mature method with a core 5-step process and supporting IT techniques RPR requires no pre-conceived idea of the cause of the problem, in fact such thoughts are positively discouraged The method uses non-disruptive techniques and so there is minimal business impact RPR doesn‟t require recreation in a lab environment, or even testing outside of normal working hours
The method is based around the collection of definitive diagnostic data at the exact point of a problem and so precisely identifies the cause; transient or not RPR‟s primary objective is to identify the root cause
RPR enhances the skills of the IT team and support companies
2
In the late 1970‟s and early 1980‟s IBM taught its software engineers a two stage process of problem diagnosis called Problem Source Identification / Problem Determination.
RPR Process RPR starts with the premise that first we must identify the root cause, and only then can we define a fix. The method differs from many Root Cause Analysis methods that start from the fix and work backwards to the root cause. RPR begins by determining the root cause and then works forward to a solution. The root cause is determined through a five-step process: Gain an accurate understanding of the problem at some level Choose one specific symptom Create an Action Plan to capture definitive diagnostic data for one or more identifiable instances of the chosen symptom Execute the plan whilst controlling the environment Analyse the results and either; - identify the root cause and determine a fix, or - define a new Action Plan and execute it It is likely that we will need to iterate around the last three steps, revising our Action Plan and re-analysing the results. At first sight the process might look ridiculously simplistic but the devil is in the detail.
Understanding the Problem How many times have you heard the complaint, “The network is slow”? What does that term actually mean? What is slow and how slow is slow? An accurate understanding of the problem might be:
When I click on the Appointments button in our calendaring system I usually see the diary for the day within 5 seconds, but intermittently it takes more than 15 seconds. Our understanding must be of a high level to enable us to replicate the problem in a test environment.
A Single Symptom RPR dictates that we can only diagnose one symptom at a time, even if we think many symptoms have one common cause. This can be a tougher proposition than you might think. The RPR Practitioner can come under considerable pressure to deal with all of the issues – particularly as “they are all linked”. RPR warns against trying to establish patterns, i.e. links between differing symptoms. There are many reasons for avoiding using a pattern method to diagnose problems and here‟s an illustration of one. The Service Desk reported that users suffered a “3 o‟clock slowdown”. At around three o‟clock every day users said that the network was slow. Three specific symptoms were identified; Outlook Inbox items were slow to open, Word documents sometimes took 30 seconds to save and Citrix users suffered intermittent type-ahead delays. It turned out that none of these problems had a common cause. Starting from a presumption that all were linked would have led to failure to find the root cause of any of them.
If multiple symptoms have the same cause then by fixing one we will fix them all.
Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
Definitive Diagnostics Shortcomings with Statistics Generating definitive diagnostics is a very big subject that alone could fill several whitepapers. The most important point of this step is that we must be able to gather diagnostics that can be directly correlated with the user‟s experience of the problem. RPR rejects the use of statistical data that cannot be directly matched to the moment that the problem occurred. This usually comes down to “wiggly graphs” like this: CPU Utilisation 100 90 80 70 60 n o i t a s i l i t U
50 40 30 20 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 0 2 : 0 4 : 0 6 : 0 8 : 1 0 : 1 2 : 1 4 : 1 6 : 1 8 : 2 0 : 2 2 : 2 4 : 2 6 : 2 8 : 3 0 : 3 2 : 3 4 : 3 6 : 3 8 : 4 0 : 4 2 : 4 4 : 4 6 : 4 8 : 5 0 : 5 2 : 5 4 : 5 6 : 5 8 : 0 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1 1 : 1
Time of Day
If the CPU utilisation of a server was constantly above 90% it would make sense to solve this, although even then such load does not guarantee that we have discovered the cause of the performance problem in a complex end-toend system. More often than not we are actually faced with a graph like that above, the interpretation of which is very subjective. th To remove the subjectivity we might work on a designated overload threshold of 50% for the 95 percentile figure but even that ignores the issues of transient problems which can get hidden by the averaging that occurs over the sample period. A possible solution is to use a more granular measurement based on one-second samples, but even then we must be able to match the start and end time of the problem to the correct points on the graph. Without correlation there is a danger that we might spend money, and, more importantly, time on upgrading the server only to find that we still have the problem. Correlation RPR proposes that we gather diagnostics that can be directly correlated with one or more user experienced problems. So returning to our earlier scenario of the slow response time to the Appointments button we might decide to set-up: A network trace for the user‟s PC
A network trace showing everything going in and out of the application server A SQL trace for all database calls A perfmon study of CPU, memory and disk I/O on the application and SQL server
We then wait for the problem and when it occurs we immediately mark the diagnostic data. Many of the Supporting Techniques of RPR are designed to address the issue of correlation of the user experience of a problem to the corresponding diagnostic events.
Markers Markers are a key RPR technique to achieve correlation. A marker is simply an entry in the diagnostic data that is generated under our control, and is unique and easily identifiable.
Here are some examples of markers: A ping with a payload length of 101 bytes – ping –n 1 –l 101 LONSERVER01 - Generates an identifiable entry in network trace data - Correlate with perfmon by adding ICMP / Received Echo / sec counter A GET for a non-existent URL – http://intranet/marker101.asp - Generates a trace entry and web log entry so that we can match server and analyzer time A dir command for a file that doesn‟t exist - Generates a marker in a filemon and / or procmon trace
Remote execution of Performance to show the Thread Count for a process every second - When the process dies we see the event in the network trace Use a SQL client to generate an identifiable query in a database – select „marker101‟; - Generates a marker in a network trace and SQL profiler trace
Execute the Plan A major consideration in the execution of the Action Plan is CONTROL. It is important that we carefully follow the steps in our plan in a very controlled manner. This too can be challenging, particularly if the problem is highly intermittent. Here are typical things that can go wrong: Delayed Notification - The user agrees to call us the moment the problem has occurred. When he does call we find that he has done “a few more things” since the problem Procedure Failure - The user agreed to send a marker immediately after the application gave a slow response, but now can‟t remember if he sent it during the problem, immediately after, or some time later Freelancing - The problem is being tackled by a large team and, despite having agreed to the plan, some members of the team think they know what the problem is and so make some changes. The changes invalidate our diagnostic data Pragmatism - IT management decide that rather than wait for the problem to occur again they will get the server upgraded
In all these cases, and many more like them we have to set the clock back to zero. We cannot safely assume that anything we have already discovered remains true. Of course, every IT department comes under great pressure to fix a problem as quickly as possible. Sometimes a pragmatic approach is needed and we are not able to follow the RPR method. We just need to make sure that everyone involved accepts that RPR will not work if ad-hoc changes are made.
Analyse the Results The RPR Supporting Techniques provide many techniques for detailed analysis of diagnostic data. Here are just a few tips: Identify the markers in the diagnostics, and hence the boundaries of the problem Filter the diagnostic data to get a high level view of the events leading up to the problem
When studying a slow response time from for example, an application server, try to account for all of the time spent on network interactions – anything left over must have occurred inside the server When studying a failure or a hang, compare diagnostics for a working scenario with those for a failure and focus on the point where things first differ Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
It is useful to consider the type of output that we would like from the analysis. Continuing with the Appointments scenario, once we have successfully executed the plan, careful analysis of the diagnostics might lead us to the conclusions that: At the time of the user reported problem we can see from network trace data that the database takes 13 seconds to execute a request The request is a stored procedure call sp_GetCustomerDetail At this time SQL server CPU load is 65%, max disk I/O queue length is 2 and there is 1.2 GB of memory in use Analysis of a matching SQL Profiler trace taken at the time shows that an additional index is required on table custinfo
RPR Supporting Techniques Overview RPR presents a very wide range of techniques to help us follow the RPR Process. In summary these are: Initial Data Gathering – getting up to speed with the problem Initiation Workshop – a structured brainstorming session Accurate Data Collection – covering the practicalities of generating definitive diagnostics Diagnostic Data Analysis – practical tips to help the analyst convert all that data into information Analysing Intermittent Failures – particular steps to aid the detailed analysis Problem Management – how to best reflect status and progress back to the business Soft Skills – how to deal with the politics and the pressure, and how to enhance the skills of others
The techniques presented are generic ie they are applicable to any technology. Although some tools and technologies are cited as an illustration of a technique, RPR teaches nothing about specific tools or technologies. One of the Supporting Techniques is Whiteboard Analysis, and we‟ll take a brief look at this by way of example.
Whiteboard Analysis We use whiteboard analysis to pull together the strands of a brainstorming session, the first of which is the Initiation Workshop. The technique is quite simple. Write five headings on a whiteboard or flipchart: Symptoms Boundaries Other Observations Possible Causes
Action Plan
Under Symptoms write an accurate description of one or more symptoms to a level that would enable us to attempt to recreate the problem. Prioritise the symptoms and choose the one we intend to tackle – remember, we can only tackle one.
Under Boundaries note when, where and under what circumstances the problem occurs. These boundaries are used to determine when, where and how to conduct the investigation. They are never used to reach a conclusion regarding the cause of a problem. The boundaries simply guide you to the best time and place to do the diagnosis. For example, if the problem has only ever been noted in the Munich office at 3 pm on a Friday, there is little point trying to collect diagnostics in London on Mon day morning. That doesn‟t mean that the problem is linked to the Munich office or the time of day. For all we know the problem might occur in London, but the users just don‟t report it, or, their pattern of application use is different. Other Observations are things that may or may not be related to the problem, and are worth noting for later consideration but must not figure in our diagnostic efforts. Put simply, they must be ignored at this stage. From the Symptoms and Boundaries you can identify the components of the end-to-end system. Each of these is a Possible Cause. Don‟t be too granular in determining the possible causes – choose large chunks of infrastructure. The list might be: User PC Munich LAN WAN Data centre LAN Application Server Database Server
Now devise an Action Plan to prove that a particular Possible Cause is, or is not, the root cause of the problem. If it‟s possible to maintain good control of the diagnostic data capture, diagnosis can be made quicker by gathering data from many places along the end-to-end path. For the first pass make sure that the plan includes capture of data adjacent the user‟s PC since it‟s here that the user experience can be most easily correlated with diagnostic events.
Appendix B - Whiteboard Exercise Briefing Sheet on page 11 gives a sample scenario for practice.
RPR Challenges & Limitations It is worth noting the limitations and demands of RPR, some of which can be difficult to overcome if all involved do not understand or believe in the method: Changes must not be made immediately prior to and whilst investigating the problem. Any change that affects the system being studied will invalidate the diagnostic data collection RPR requires the users to suffer the problem at least one more time. The method is totally evidence-based and that means collecting the diagnostic data when the problem occurs RPR can only deal with one symptom at a time. This is often controversial as there is an incorrect belief that this is a slower method than taking an all encompassing approach to determine a common cause
IT Management must allow the Problem Analyst dedicated time for most of the duration of the problem analysis – certainly during the data collection phase The Problem Analyst must have a good knowledge of diagnostic tools, when to use them and their limitations The Problem Analyst must have strong analytical skills – I guess this goes with the territory Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
Initially selling the approach as the most effective method of resolving Phase 4 problems can be tough, but, on the positive side, it becomes easier to promote the method after the first success.
When to Use RPR RPR is not appropriate for the majority of problems that arise – certainly not for Phase 1 and 2 problems (or events that ITIL classifies as Incidents). However, it is a very effective and efficient method for the resolution of Phase 4 problems. A Phase 4 problem will have some or all of the following characteristics: Senior IT management involvement Business demands regular status updates Regular crisis meetings Help Desk calls about the problem drop off Business adjusts its work processes to accommodate the problem An ever growing group of people are getting involved There is no sense of direction in the problem resolution effort The root cause is unknown Repeated statements are made that “We just want to try one more thing”
The final point is an ideal trigger for a switch to RPR, as it indicates that resolution has moved into a “trial and error” phase which is very slow.
Skills Needed A broad knowledge of IT is needed to make best use of RPR. The practitioner must have a good basic knowledge of the technologies and concepts of an end-to-end system. We live in a networked world and modern systems have many components networked together. The networking slant is further influenced by the choice of tools. The natural tool of choice for analysis of end-to-end problems is the network analyser since: Its use is non-disruptive There is no requirement to install additional software on any components of the system It can be used to narrow a problem to a component, and hence an owner
This means that a good knowledge of networking and protocols is a significant advantage. Once we have narrowed the problem to say a server, we may then need to use further tools to drill into that component, and this demands further skills, largely based around knowledge of software and operating systems. For career enhancement, becoming an RPR practitione r is ideal for senior IT support staff who don‟t want to move into management, but need a new challenge. A very bright junior would also be a valuable addition to a problem
management team but it‟s important to note that there is often a lot of politics , many business considerations and significant people management in the resolution of a Phase 4 problem, and so senior support is likely to be needed.
The ideal RPR person would be a senior application developer who has a good knowledge of networking – or vice versa. Those companies that have embraced ITIL may have dedicated Problem Managers and Problem Analysts. RPR is ideal training for both since the RPR Process provides a framework for the Problem Manager, and the Supporting Techniques help develop effective skills in the Problem Analyst.
Problem analysis skills can be taught (as per RPR) and require development through experience. It‟s important to recognise that effective problem analysis requires its own set of skills. These skills are often built upon operational (BAU) experience, but the skills are different.
For a Problem Analyst to fix problems quickly with RPR, he or she needs to be exposed to Phase 4 problems almost continuously. It may be difficult to keep a Problem Analyst sufficiently occupied in an organisation of less than, say, 8,000 users.
Closing Comments I hope this short introduction has given some idea of the power of RPR. There is only so much that we can cover in a whitepaper. However, you can benefit today from this whitepaper by: Using it as a guide, you could set an escalation point to recognise that a problem has entered Phase 4 and requires a different approach – hearing or using the phrase “We‟re just going to try one more thing” is a good starting point Focusing on one symptom of a Phase 4 problem will simplify diagnosis and speed up resolution Avoiding pattern-based methods will increase the likelihood of success Collecting definitive diagnostics will identify the root cause, which will save time and money Using the Whiteboard Analysis will help you plan the diagnosis
Build your knowledge base Rapid Problem Resolution (RPR) Paul Offord th 4 September 2007
Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
Appendix A – Outlook Freeze Case Study 3
This was a very recent REACT™ project and perfectly demonstrates the collaborative nature of RPR. Users complained that intermittently Outlook would freeze. The first issue was a plethora of symptoms that were believed to be due to the same cause. We asked the customer to prioritise one symptom, and settled on this problem scenario: The user opens an Inbox message and reads it He opens a Word document that resides in a Document Management System folder visible in Outlook – the DMS being integrated with Outlook He makes a change to the document and closes it He tries to switch back to the Inbox message he was reading but just gets an empty window (all white) Around 75 seconds later the message window fills with the missing text
Using RPR we started by determining if the problem was inside the user‟s PC, or, whether the network or servers caused it. To do this we traced all network data going in and out of the user PC. We then set up a small command line program to send pings of different lengths when the user hit Return. When the problem started we sent a single ping of 101 bytes. When the problem passed, and as soon as the Inbox message became visible again we sent a second ping of 102 bytes.
Analysing the trace we found the 102-byte ping and wound back through the trace, starting at this point. It was evident that the probl em passed after the completion of the transfer of 15 MB of data from the DMS server to the user‟s PC. A study of the data showed a repeating pattern that looked like a list of documents. We took the data to the DMS support team who immediately recognised it as folder summary information. Working with the DMS team, we identified it as cache refresh information. A call to the DMS vendor quickly identified the issue as an incorrect registry setting. This scenario is very typical. Although we (the RPR pract itioners) had some knowledge of the DMS, we didn‟t know as much as the support team let alone the vendor. Following the RPR method caused us to: Focus on a single symptom Collect high quality diagnostic data that could be directly correlated with a user‟s experience of the problem
Convert the diagnostics into information that is meaningful to the DMS support team
This was a text book RPR scenario and demonstrates the collaborative nature of the method. I hope it also shows that you can use the method even if you are not an expert in the particular technical subject.
3
REACT™ is Advance7‟s unique method-based problem resolution service.
Appendix B - Whiteboard Exercise Briefing Sheet Cambridge
Southampton
Derby
User PC Giant Finance WAN
Internet
User PC Citrix Server
User PC
Staff at Giant Finance use a third-party application called TopFund hosted by Acme Corp. Access is via a Virtual Private Network (VPN) e stablished across the Internet from Giant‟s Southampton site to Acme‟s Derby Data Centre. Giant Finance staff at Cambridge say that the system is so slow that it is unusable. Here‟s the information as it is told to you:
Only Giant Finance users in Cambridge suffer the problem, so it must be a Giant Finance problem Cambridge users don‟t have problems with any other systems and so it must be an Acme problem No other Acme users get the problem and so it must be a problem with Giant‟s VPN firewalls
Giant users access other 3rd party systems via the same VPN firewalls and they don‟t have problems 4
Users of the Citrix-based TopFund system suffer type-ahead delays The problem only happens when more than one Cambridge user accesses TopFund so it must be a Citrix server issue Lots of users access the same TopFund servers and don‟t experience problems
All Giant users experience slow web access but the Citrix traffic has been prioritised above web traffic
Write five headings on a whiteboard or flipchart: Symptom – comes from the rule “You can go no further until you understand the problem at some level”. It must be accurate. It must be specific Boundaries – used to help you decide where and when to investigate the problem Other Observations – things that may or may not be related to the problem Possible Causes – at a high level Action Plan – a plan to prove or disprove the first of the possible causes
4
Users of terminal-type services that use character echo protocols such as Citrix ICA, Windows Terminal Services and TELNET sometimes complain that they type characters but nothing appears for a few seconds, then all the characters appear at once. We call this effect type-ahead delay. Executive Whitepaper © 2008 – Advance Seven Limited
Advance7 Defining IT stability, control & performance
www.Advance7.com +44 (0) 1371 876805
Appendix C – Whiteboard Exercise Answers Symptoms Users of the Citrix-based TopFund system suffer type-ahead delays Boundaries Only users in Cambridge experience the problem Problem only occurs when more than one user accesses the system No other TopFund users get the problem Other Observations Cambridge users don‟t have problems with other systems except for slow web browsing VPN connections are made to other systems and they work fine Possible Causes Client PC Cambridge LAN Giant private WAN Southampton network infrastructure Internet Derby network infrastructure Citrix servers Action Plan Install an analyser in Cambridge to capture traffic in and out of the user PC Install a second analyser in Cambridge to capture traffic in and out of the WAN router Install an analyzer in Southampton to capture traffic in and out of the WAN router Recreate the problem, marking the traces with a ping immediately after the problem occurs Analyse and match the traces to determine if the problem is in the user PC, in the Cambridge LAN, in the WAN or between the Southampton LAN and the Derby server Produce a revised plan if necessary
If this were a general complaint of slow performance, we might make “Slow web browsing” a symptom and then determine the priority of the two symptoms. However, in this case the issue at hand was Citrix type-ahead problems, and so the slow web browsing is relegated to an observation.
