SAP MANUAL TESTING HOW TO TEST LESS, FASTER, AND BETTER
CONTENTS Introduction .......................................................................................................................... 2 Why is Automated Testing Such a Challenge for SAP ERP? ................................................................... 2 SAP Manual Testing is Underserved .............................................................................................. 3 Deciding What to TEST: Cost vs. Risk ......................................................................................... 4 Creating Test Scenarios for Scripted Testing ............................................................................... 4 Unscripted “Exploratory” Testing ............................................................................................. 5 Capacity of Business Key Users ................................................................................................ 5 Documenting Tests Performed ................................................................................................. 5 Managing the Manual Testing Process ........................................................................................ 5 Optimizing the Manual Testing Process.......................................................................................... 6 Automated Test Scoping of SAP-Driven Changes ........................................................................... 6 Accelerated Creation of Test Scenarios ...................................................................................... 6 Accelerated Execution of Manual Testing ................................................................................... 6 Accelerated documentation of test runs ..................................................................................... 7 Designed and Built for Business Users ........................................................................................ 7 Test Management ................................................................................................................ 7
-1-
INTRODUCTION SAP ERP systems must evolve at the speed of business change. It is therefore imperative for SAP ERP managers to adapt their SAP landscapes in a timely manner to address business needs such as process changes, rollouts to new sites, and regulatory updates. In addition, they must cater to the maintenance needs of their SAP systems and implement SAP support packs and enhancement package upgrades on a regular basis. SAP ERP systems are highly complex, built on hundreds of millions of lines of standard code and customizations. This means that even a small change to the system could trigger a series of inadvertent consequences. To prevent production failures, companies regularly perform a series of functional test cycles – which can be either manually or automatically executed. Industry papers discussing approaches to test automation have been available for well over a decade. Little has been discussed, however, about the applicability of test automation to SAP ERP. This document describes the specific challenges of SAP ERP functional testing, when test automation is inappropriate, and how to perform manual testing faster, better, and cheaper in these cases.
WHY IS AUTOMATED TESTING SUCH A CHALLENGE FOR SAP ERP? A survey performed by Panaya in 2012 of 100 SAP-run enterprises revealed that even the most mature testing organizations perform only 15% to 25% of their functional regression testing using test automation tools; the remainder is manual. There are three main reasons, inherent to SAP ERP, which explain why test automation coverage is relatively low: (i)
(ii) (iii)
SAP ERP functionality is heavily data driven. SAP business logic is dependent on and functions differently with change in the input data; this translates to changes in screen flow, screen content and screen format that must be f actored into the test automation script. For business processes that change frequently, the test automation scripts must be continually kept up to date. A challenging and costly proposition. Preparing the required test data for test automation is also challenging due to SAP’s complex data structure and the various dependencies that exist in an SAP business process. Think about an automatic test script for a purchase-to-pay process that requires five different material types for full testing coverage. Maintaining integrity of the data required to support the end-to-end process for the five material types is a daunting task.
-2-
SAP MANUAL TESTING IS UNDERSERVED Since it’s virtually impossible for a company to develop automation test scripts to cover every possible SAP business scenario, manual testing is heavily relied upon to ensure that all the needed functionality is covered. Yet manual testing is severely underserved in terms of tools that increase productivity, enforce processes, and provide control. Performing manual tests requires an intimate understanding of the business processes to be tested, and so, it is typically the functional experts in IT or key business users (sometimes referred to as super users) that are tasked with performing manual testing. They do their best to test, report defects, report progress, and document test runs, on top of their other daily duties. This is all performed manually, with tools such as Word and Excel. In organizations where a dedicated test manager is allocated to the SAP team, management of the testing cycle is performed by the test manager. In the majority of SAP shops, however, there isn’t a dedicated test manager; the test manager’s role is performed by project managers, in the scope of their respective projects, as an added duty. Test management includes defining the testing strategy, identifying the tests to be performed, assigning tests to the manual testers, chasing the testers for their individual status reports and consolidating them to a “bigpicture” testing progress and coverage status, managing defects, ensuring that testers perform the testing on time, and more! Approximately two thirds of SAP shops use general productivity software like spreadsheets in support of these test management activities. It is little surprise, therefore, that they lack visibility and control over the testing cycle, and put too much time and effort into managing the testing activities.
-3-
DECIDING WHAT TO TEST: COST VS. RISK The first pain point comes early in the testing cycle: determining what, exactly, to test. Following a technical or functional change in a massive application like SAP ERP, it’s a challenge to determine where exactly in the system testing should be focused. Customers have two options. One is to try to test every possible SAP transaction that may be affected to minimize risk of production failures. That, of course, increases testing time and cost. The other alternative is to essentially ‘guess’ where the greatest impact will be and test those areas. The second option comes with lower costs but much more risk. In practice, most organizations seek a happy medium between the two extremes, but the costly nature of manual testing makes many lean toward less testing. Finding a method to test more functionality within a strict timeline is one of the biggest challenges associated with implementing changes in the ERP application, so finding a cost effective resolution is extremely v aluable.
CREATING TEST SCENARIOS FOR SCRIPTED TESTING Test scenarios describe the steps needed to test a given function or process and the anticipated results of every step. In theory, this should work well: each manual tester has a written script telling him or her what to do, how to do it, and what results to expect. But in practice, this is far from the case. Typically written in Word or Excel, test scripts contain varying levels of detail, with no standardization as to what should and should not be included. It’s a time-consuming process, as each script can take from 30 minutes to several hours to create. And there’s no guarantee that others will be able to reliably follow the script; what’s clear to whoever wrote the script may not be to those charged with following it.
-4-
UNSCRIPTED “EXPLORATORY” TESTING In some cases, test scenarios don’t exist at all. Instead, organizations practice a form of ad-hoc testing, known as exploratory testing, where manual testers are tasked to perform a test as they see fit and report results. There are two main use cases for exploratory testing: 1. The company doesn’t have pre-defined scripts and there is no time to create them. Therefore, manual testers run a test cycle based on their knowledge of the way the application is used. While this reduces test preparation time, it adds risk that the business process will not be tested adequately. 2. The QA team or functional testers create test scripts and conduct tests based on these scripts. At the User Acceptance Test phase the test manager doesn’t want the key user to repeat the exact same scripts, because they were successfully tested already. Furthermore, the test manager needs to ensure the application is tested exactly as the business users will use it in their daily work, so he asks the user to simply do their day-to-day tasks on the QA system and check that everything works properly. This method provides an additional assurance for the quality of the application. Problems may arise when exploratory testing uncovers issues that developers must address. With no record of what, exactly, the tester did, it’s difficult for the developers to accurately recreate problems that need fixing. What’s more, there’s no knowledge transfer of the testing process from the business user to IT to enable IT to create test scenarios for future use.
CAPACITY OF BUSINESS KEY USERS Testing takes the business key users away from their pressing day jobs and limits their ability to fulfil their other duties, such as supporting end-users, delivering training, defining processes, or liaising with IT regarding functional needs. These key users are typically among the most experienced and highest paid in the user organization and have very little time to spare. Since the key users are forced to fit test activities around their normal workload, testing often has to wait.
DOCUMENTING TESTS PERFORMED At the same time, these time-strapped key users are often required to collect evidence to prove whether or not these tests were successful. This is especially true for organizations subject to government regulations, such as Sarbanes Oxley (SOX), U.S. Food and Drug Administration (FDA) rules, and others. Users must collect detailed proof of what was tested, by whom, and how. It can be a painstaking, time-consuming process for testers to constantly take screen shots, then copy and paste them into a document to show which tests were performed. This process can take at least 20 to 30 minutes per test, which translates into several days of nonproductive work per test cycle per key user
MANAGING THE MANUAL TESTING PROCESS Finally, when testing is manual, the testing manager has no real-time visibility into the process. He can distribute detailed instructions to testers but has no way of knowing if, when, and how they are conducting the tests. As a result, he can’t unequivocally know the level of progress being made and the precise issues which have cropped up. When testers are located at multiple sites, the challenge becomes all the greater, forcing the manager to resort to emails, spreadsheets and phone calls to keep tabs on testers.
-5-
OPTIMIZING THE MANUAL TESTING PROCESS So it’s clear that manual testing will always be required; there’s simply no way around having experienced users verify that the SAP system functions as it should. However, many aspects of the testing process can and should be accelerated and managed. These include: • • • • • •
Determining what to test (test scoping) Creating detailed test scenarios Executing manual tests Reporting defects that can be easily reproduced Documenting test run evidence Tracking test run progress
In short, manual testers require better tools to make the process more efficient, accurate – and less painful. Testing managers need visibility and control over manual testing cycles. This visibility will enable organizations to conduct more testing in the same time window and hence, find more problems before the code goes into production. Increasing testing efficiency also has the effect of driving down the cost of testing and potentially speeding up project implementation, thus increasing business agility.
SAP IT t eams seeking a solut ion t o accelerat e the dif f erent f acets of t he manual t est ing process should e v a l u a t e m a n ua l t e st i n g so l u t i o n s o n t h e b a si s of t h e f o l l o w i n g a t t r i b u t e s :
1. AUTOMATED TEST SCOPING OF SAP-DRIVEN CHANGES Determining which functionality of the SAP ERP to test following a major change – especially when implementing SAP-driven changes, such as support packages, enhancement packages or business functions – is crucial to minimizing the risk and cost of change. Analyzing the production usage of your ERP transactions combined with an impact analysis of the changes on your system can yield a risk-based approach to test prioritization that can reduce testing efforts by as much as 70%.
2. ACCELERATED CREATION OF TEST SCENARIOS Writing test scenarios is a painstaking process but it’s also one that’s well suited to automation. Your testing tool should capture the business process to be tested while a functional expert or key user navigates the ERP application. The testing tool should translate the steps into structured human-readable test scenarios, including documentation of expected results for every step of the scenario. You’re essentially capturing the knowledge of that key user and making it available for all others to emulate in their own testing. At the same time, you ensure the scripts are highly accurate – but take far less time to create.
3. ACCELERATED EXECUTION OF MANUAL TESTING Similarly, your testing solution should ease the burden on testers by offloading many manual tasks associated with testing, such as keying the right data into the transaction fields and comparing the results to those expected, to ensure accuracy.
-6-
4. ACCELERATED DOCUMENTATION OF TEST RUNS Another aspect of test execution is the process of capturing and documenting test evidence for every test run. Seek a testing solution that enables you to determine exactly which process the tester followed, what the results were and how those results compare to what was expected. This is a crucial step in terms of providing proof of testing for auditors, as well as tracing production failures to the gaps in your test scenarios. It also relieves testers of the tedious process of manually documenting test results.
5. ACCELERATED REPORTING OF DEFECTS The ability to effectively report functional failure is crucial in enabling IT address issues uncovered by testing. Developers should get an accurate, screen-by-screen and keystroke-by-keystroke depiction of how the tester reached a problem point, so they can recreate and fix the problem. This is exactly the kind of data that testers struggle to capture and communicate accurately.
6. DESIGNED AND BUILT FOR BUSINESS USERS Remember your key business users will ultimately have to use the testing tool as part of their daily routine. They are already pressed for time; the last thing they want is to spend time learning how to use a testing tool that they never wanted to use in the first place. So they need seamless access to the testing tool from the ERP application. It should require minimal, if any, training to perform such testing activities as: viewing the assigned test tasks, executing the test tasks, testing in collaboration with other testers to complete a full business process, and reporting defects and progress. The tool should not require complex installation of clientside software, and must be intuitive and simple to use, without interfering with the non-testing tasks of the key user.
7. TEST MANAGEMENT Finally, your manual testing solution should provide y ou with ongoing visibility and governance over the end-toend testing process. You should be able to plan test projects, assign tasks to all manual testers, determine how close the testers are to completing their assigned tasks, and receive alerts when problems arise or steps are completed. The solution should also enable you to track a distributed multi-site testing operation.
8. SIMPLE SET-UP, MINIMAL ONGOING MAINTENANCE You need a solution that doesn’t require cumbersome software installations, frequent patching or painful upgrade processes on the manual testers’ client machines. It should take IT less than a day to set up the full solution.
KEEP UP THE PACE The pace of business change isn’t likely to slow down, so you can’t expect the SAP ERP testing burden to diminish either. Organizations that take the initiative to optimize t heir manual testing processes will ease the burden on manual testers, enabling them to spend less time testing while allowing IT to complete more projects. This ultimately empowers the company to become more agile and competitive. Learn how Panaya can help you optimize your SAP testing processes. Visit us at: www.panaya.com.
-7-
DISCLAIMER AND TRADEMARK NOTICES This report is provided by Panaya Ltd. It is completely independent of and not affiliated with Oracle or SAP AG. Oracle is a registered trademark of Oracle. Oracle and other Oracle products and services mentioned herein a s well as their respective logos are trademarks or registered trademarks of Oracle. SAP is a registered trademark of SAP AG. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. All other product and service names mentioned are the trademarks of their respective companies.
DISCLAIMER OF WARRANTY Panaya Ltd. makes no representation or warranties, either express or implied by or with respect to anything in this document, and shall not be liable for any implied warranties of merchantability or fitness for a particular purpose or for any indirect special or consequential damages.
COPYRIGHT NOTICE No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, photocopying, recording or otherwise, without prior written consent of Panaya Ltd. No patent liability is assumed with respect to the use of the information contained herein. While every precaution has been taken in the preparation of this publication, Panaya Ltd. assumes no responsibility for errors or omissions. This publication is subject to change without notice .© 2013 Panaya Ltd. All rights reserved.
ABOUT PANAYA Panaya SaaS Solutions empower companies that use SAP or Oracle to reduce up to 70% of their upgrade and testing risk and effort. Panaya simulates the u pcoming upgrade, automatically pinpointing which custom programs will break as a result of the upgrade and automatically fixing most of these problems. Panaya’s testing solutions dramatically expedite ERP testing and eliminate the need for manual test script maintenance. Seamlessly capturing business knowledge in the background, as use rs work with the ERP applications, Panaya automatically generates plain-English test scripts that are rapidly executed and continually self-adjust based on test results.
-8-