Game Testing Methodology 1.
INTRODUCTION
Effective testing comes from a well-structured approach and a well-defined testing methodology so the game product is highly highly satisfyi satisfying ng to our Publisher Publisher and the game player player.. In contrast, poor po or testing results in a buggy game or software that gives rise to a long stream of repeated testing and project delays. delays. The purpose of this document is to explain the testing methodology methodology for game development. development. This document is intended for the person performing the testing . The topics and the level of details provide an interesting interesting and useful set of information information for the ame Tester Tester or !oftware Tester. Tester. There are a number of "uality #ssurance and testing initiatives that are covered in this document, includ including ing a testing testing strategy strategy,, testing testing processes processes and techni$ue techni$ues, s, test coverage coverage criteria criteria,, Testi Testing ng %e$uirements, a Test Plan definition, testing documentation, and &'ull (ycle Testing). There are few, few, if any &fixed &fixed rules) rules) for this this testing testing methodolog methodology* y* however, however, there are many many suggestions, ideas, and guidelines for improving the $uality and effectiveness of testing for our game projects. +opefully, testers will be able to learn, understand, plan and carry out effective and efficient testing in a structured manner. If you have any $uestions about the information, please contact your !oftware "uality "uality irector. This document must be used in conjunction with the other companion "# documents . ame ame Tes Testi ting ng Pri Prime mer, r, . Testin Testing g esig esign n for for a game game project, project, and and /. efect efect %epo %eporti rting ng uid uidel elin ine. e.
ame Testing 0ethodology
2.
TESTING STRATEGY
2.1
SCOPE AND DEFINITION
In a simplistic view, testing is to identify bugs found in the software , so the problem can be removed. There are different forms of tests and testing that can be categori1ed as “Black-Box” testing and “Clear-Box” testing 2&(lear 3ox) testing is also 4nown as &5hite-3ox) testing in the software industry6. Their testing objective and overall processes are indifference 2e.g., test planning, test design, testing execution, regression testing and bug reporting6, but their focus of attention puts emphasis on different aspects of the game7
&Black Box) focuses on the functional or the playability aspects of the game. 'or examples, testing the user interface 2e.g., the selection menus and the use of buttons6, the &loo4 and feel) 2e.g., the graphics and animation6, and the actual gameplay. &Clear Box) is focus on the architecture and integration aspects of the game software. 'or examples, the use of a database, pipelines, the interaction8integration of game components li4e the rendering engine, the #I engine, sound, and so on.
'or 3lac4 3ox testing, the tester must 4now how to play the game 2e.g., use of the game pad, 4now the rules and the game flow6. 'or (lear 3ox testing, the tester must understand what coding is. The !oftware Tester uses a run-time debugging environment, feeds the code or chun4s of code with input 2i.e., data, setting variables, etc.6 and analy1es the test result.
2.2
TESTING PHILOSOPHY
Testing is NOT a single person9s job, nor solely the responsibility of the ame Tester and the !oftware Tester for a game project. Every team member wor4ing in a game project must have &$uality) in mind, and each person is responsible for the accuracy and completeness of the wor4 that he8she produces. This testing methodology is NOT the only process and it should NOT be used in isolation. The reader must be aware that this testing methodology is considered as an integral part to the ame Pre-production and Production processes. In reality, no one can test a program COMPLETELY, i.e., testing every single part of the game using different and all available characters, so triggering different path of the logic and all the possible variations of input, interfaces, #I, and then output. :ur testing strategy is to develop excellent, full-coverage, and effective testing 2i.e., ;<8< rule6.
ame Testing 0ethodology
.
TESTING PLANNING AND TESTING RE!UIREMENTS
.1
IDENTIFY THE TESTING RE!UIREMENTS
# re$uirement is an objective that must be met. The tester must understand most of the game re$uirements and translate them into functional terms 2e.g., what needs to be tested, what things are testable and what not, what are the targets and measures, etc.6, leaving the design and implementation details to the developers. #s part of the testing re$uirement development process, the testers read the system documentation, gather all the visual and written information, analy1e what and how the game components can be tested. It is the responsibility of testers to read all the relevant documentation so they can share (to understand and appreciate) the mission of the project. Please refer to the &'ull (ycle Testing) section about how to conduct a $uality review. =ou are re$uired to develop a Testing %e$uirements document for each game by outlining what and how the game and game components will be tested. The document includes7
a list of features, the specifics of the internal design and external designs of the game. This may re$uire a description of the possible implementations if it ma4es the testing re$uirements easier to understand 2e.g., certain theme of the game, the characters, the animation, the #I, cinematic or camera view, and so on6. 'or example, to test the multidirectional fight action for the (han P! game, you must ma4e reference to the use of &%ing of Engagement), describe how the opponents engage into the fight scene, and what you expect the single8combination fighting actions. a testing structure detailing if and how ame Testing and8or !oftware Testing is applied 2i.e., in a spreadsheet format for the items identified above6, the testing criteria 2e.g., ideas for testing6, and the completion criteria 2e.g., what are the expected results, what does &something is done) or &something is wor4ing) mean to you in game testing>6
#fter the testing re$uirements are identified, the Technical irector for the game and the !oftware "uality irector will review the document to confirm scope and priority. 'ollowing the test re$uirements, the testers wor4 on their own test design and develop a Test Plan and Test (ases. #ny testing dependency re$uirements must be identified and communicated to the game team so the game code is &test-enabled), i.e., what 4ind of cheat codes, or &test automation-enabled) code are re$uired> The testing documentation is expected to be developed in the early stage of the project, i.e., draft testing documentation is produced when we have the first playable build. It is important to note that the testing re$uirements will not cover every single detail of the game, but it must cover testing all the contractually re$uired elements 2e.g. specific features and the major game functionality6. =ou can obtain this information from the game Technical irector.
ame Testing 0ethodology
.2
DEFINE THE TIMELINE RE!UIREMENT
5hen a game project is pressed for time, we must recogni1e the existence of a threshold point where sufficient timeline must be provided for the testers to perform7
a number of iterations for testing each new or u pdated game features, a complete cycle of regression testing for each build, sufficient regression testing of the previously (ritical, (losed bugs, and a full regression testing expecting to test every event8world8environment object and triggers in the game for #lpha, 3eta and 'inal.
This &"#re$#ol%) point varies from game to game, the tester is expected to communicate the &bare bone) testing re$uirement with the Producer and8or Project 0anager.
ame Testing 0ethodology
&.
GAME TESTING AND TESTING TECHNI!UES
The ame Testing Primer document describes a few aspects and focuses of game testing. This section outlines a systematic testing approach, and explains how you brea4 down a game feature into its smallest testable components7
&.1
GAME COMPONENTS AND BREA'DO(N STRUCTURE
S)$"e*a"+c game testing means examining every part of a game. These parts include7
the menu and the menu functions, art 2character model, texture, terrain or world, crowd, objects, etc.6, animation 2the li4e and feel of the movement, realism, frame rate6, sound and the sound effect 2in conjunction with the facial animation, e.g. lip sync, and the animation se$uence6, music, camera 2cinematic view, 1oom in and out, replay6, game flow and logic, world8level8scene, the player attributes, the action attributes, the condition to advance to the next level 2what are the rules>6 the use of environmental objects, the event8object triggers, and the scoring progressive levels of difficulty, the #I logic 2for both defensive play and offensive play* player movement and positioning6, statistics 2pre-game and in-game li4e player statistics and high score6, title screens, ?I! 2?on-Interactive !e$uence6, !'@ 2!pecial effect6 any movie clip, the game pad, the use of multi-button actions 2also called button mashing6, the ease of use of the button functions, the shoc48vibration effect of the game pad, the use of digital and analog mode legal text, and the game options 2game start8menu selection, hints, game pause, pause menu options , and scrolling, i.e. cycling through the available options on the screen, etc.6
ame Testing 0ethodology
&.2
GAME TESTING TECHNI!UES AND ,TIPS-N-HINTS
+ere are the specific details and the testing techni$ues7
0ust examine the entire screen and not just a small part of it. 'amiliar with the game rules and test the gameplay against these rules. Test for clipping 2two or more polygons or polygon objects overlapping each other or canceling each other out6. Examine the overlap 2where a semi-transparent object shown on top of another solid colour object and8or the bac4ground6, and chec4 if the overlap is appropriate in terms of si1e, placement, the purpose and the information that is provided6. Test for incorrect and inappropriate collision 2the condition when two objects should be colliding6. Imagine when two cars collide, they should get bumped by each other with about the same effect 2e.g. damage6. In terms of physics, when a collision involves two objects with the same mass, they should generate the same amount of momentum in the opposite direction. In contrast, when a plane hits the ground, should only the plane show the effect of collision because the mass of a plane is trivial compared to the mass of the earth. 0ove the character through all the available objects and all the levels and closely examine the effect 2it could be collision, an event trigger, or an object trigger6. Test for grayed out or grey out 2when an option or icon cannot be selected6 Test loading8saving from a game save device 2e.g. hard drive or a memory card6 and ensure the correct messages are displayed on the screen. Ensure a &game load) or &data load) message and an appropriate &loading) indication 2e.g. loading meter6 displayed on the Aoading !creen. Ensure the loading time is acceptable 2no one li4es to wait for more than < seconds6. Aoo4 for micro-pause where slight pauses occur in the game but do not actually cause a full free1e or crash. 5hen it happens, it will temporarily impede the gameplay. Test for the multi-player mode 2the game runs on one machine, not on-line gaming6 where two or more players engage in one game. Test the game for memory lea4 or memory overload by leaving the game on8running for a few days. Test end of bound 2also 4nown as end of the world or edge of the world6. The game should not allow the character to move &out of bound), and8or the game becomes erratic 2e.g. game crash8free1e, the texture becomes deteriorated, the character gets trapped in certain area of the map6. Test for platform compatibility. Platform testing re$uires some extra effort for the P( game and on-line gaming7 'or a P( game, test for all the supported operating systems li4e 0icrosoft 5indows BC8BD8B;8<<< and ?T 2this includes Install, ninstall, and actual gameplay6. #lso, test for different sound8graphic cards and peripherals that are available in the mar4et 2discuss with your Technical irector for the ma4e8model of the peripherals6.
ame Testing 0ethodology Test the networ4ing for game play over modem2s6 at different speed, and Internet gaming (The depth and coverage for Netork Testing ill expand if the ga!e itself relies on so!e "ever applications for the actual ga!e pla#$ e%g% !ulti-pla#er !ode)% If the game is to be published in European countries, test for P#A conversion 2currently the game we produce is mostly targeted for ?orthern #merica, and we use ?T!(6. Test for locali1ation 2foreign language re$uirements6 if re$uired.
This is a list of common testing techni$ues used by a ame Tester. This is not meant to be a complete list, an experienced ame Tester will tell you that you should use your mind F this is where a little creativity, a little imagination, and a little bit of &common sense) become very useful.
&.
SPECIAL CONSIDERATIONS FOR GAME TESTING
3efore I let you go, I must warn you about (rac4 bugs and Placeholder, they are often mista4en as real bugs7
Crack /0$ 2i.e. a bug which in fact is NOT a bug, but someone9s cra1y imagination6. Place#ol%er 2i.e. something &dummy-up) on the screen, it is used to fill a spot until the correct video, artwor4, sound or object is implemented and incorporated into the game6.
ame Testing 0ethodology
.
TEST PLAN DE3ELOPMENT FOR GAME TESTING
There are two types of test cases* one is testing the functionalit# of the product* commonly 4nown in the software industry as Po$+"+4e Te$"+5 . The other one is testing the durabilit# and the level of tolerance resistance of the game* 4nown as Nea"+4e Te$"+5 2it is also 4nown as !tress Testing or Aoad Testing6. &?egative) test cases are created with &brea4ing the game) in mind, so we will test for certain odd situations and determine if the game 4nows how to respond, and reacts correctly. Examples of this odd situation are,
Aoad a game without a 0E0:%= card, or pulling out the 0E0:%= card during game loading. %unning a game for more than G; hours in order to test for memory allocation and management. 5hen the game component does not handle the use of memory properly, some of the memory may become unusable 2the memory does not get recycled6. 5hen it happens, a game will become non-playable 2e.g. free1e6 after a long-playing time. 'or the playability testing, say, a scenario is developed for the &+ead-to-+ead) mode in !nowboarding, we have Player- boarding very fast, and finishing much earlier than Player-, which is boarding extra slow. 5e expect the game should be able handle this extreme situation 2e.g. no free1e, the scoring continues to wor4 the way it should be6 till Player- is finished.
efine S*oke Te$"+5 to test a new ( burn. The name of !mo4e Testing comes from the engineering lab in testing new car engines. 3efore a new engine is put out for a road test, the engineers would simply start the engine and see if it wor4s. If smo4e comes out from the engine, they would 4now immediately that some parts do not wor4 properly. Hust li4e creating a new build, we simply run the game, if the game 4eeps on crashing, we would 4now some parts are not right. The !mo4e Test for a new ( burn needs to be defined. The test plan is a simple list of what main game features and options need to be tested so you are able to confirm a new build is successful. Hust li4e testing a new car engine in the lab, you would start it and let it run for a few cycles.
ame Testing 0ethodology
6.
SOFT(ARE TESTING AND TESTING TECHNI!UES
6.1
SOFT(ARE TESTING PROCESSES
The ame Testing Primer document outlines the role of software testing. The processes that are commonly used for !oftware Testing are7 A7 Co%e I5$8ec"+o59 review the source code and focus on logic structure, conformity with standards, the use of comments, and the completeness of program documentation and8or definition. B7 I5cre*e5"al Foc0$ Te$"+59 test chun4 or pieces of code by feeding values8input to an isolated module, and analy1e the intermediate result from the module. sually a run-time facility, a debugging tool, a &dummy) front-end, which allows invocation of objects by their underlying method2s6. C7 Da"a A5al)$+$9 examine the data in the database* or analy1e which part of the system uses or modifies any item of data. 3y tracing data items through the game play, the tester can validate if data usage, interpretation and manipulation by the modules or objects are appropriate. D7 Pa"# a5% Flo: "e$"+59 verify if the correct se$uence of objects or components is executed for an end-to-end path. E7 Alor+"#*-S8ec+;+c "e$"+59 test a specific game scenario or feature by setting 2or twea4ing6 data variables, data values to the code, and execute it in the run-time environment. ifferent from the Incremental 'ocus Testing, the testing is to verify a suite of codes that ma4e up a specific game feature. F7 Ar"+;+c+al I5"ell+e5ce A5al)$+$9 generate the run statistic of the programmable moves and plays of the #I components. The result is to validate if all the pre-programmed moves 2e.g. side grip on the snowboard6 and plays 2e.g. Hac4ie (han9s combination punch84ic4 in a multi-directional action6 are used. erify if the #I logic is appropriate and reasonable 2e.g. in a bas4etball game, it would not be reasonable to have the scoring B
6.2
SOFT(ARE TESTING TECHNI!UES AND ,TIPS-N-HINTS
The following provides a brief overview of specific tas4s and techni$ues that a !oftware Tester should learn or 4now while wor4ing on a game project. 17 F+le $"r0c"0re$
# new software tester should get familiar with %adical9s file structure. There are several good documents pertaining to this from the global game and foundation technologies technical documents. The learning objective is basically to learn the overall game component architecture, and understand how it all flows together and to familiar yourself with the file dependencies of the game. 5hen chec4ing a game module, it is important to understand what parts its inner wor4ings
ame Testing 0ethodology are dependent on. This will help you and speed up your bug investigation and problem analysis process. 27 Make F+le$
# software tester will be responsible for the nightly builds. Every night the build is built using the latest modules so that in the morning everyone can be wor4ing on the same up-to-date code. :ccasionally some thing will go wrong with the nightly build. # file should generate when the build is bro4en and it should say what part it failed on. =ou should use this file to in your problem investigation. sually some of the code may not wor4 with another new part or maybe the pipelines are messed up. It is the software testerKs responsibility to go through the bro4en module and to find out why it9s bro4en and fix it if possible. If its too much for you to fix, you should then inform the person responsible for the code or the pipeline that something is bro4en and get them to fix it. :nce the problem is fixed the build should be rebuilt to verify. The build number should be automatically amended with a postfix to represent the build number of the day. These build numbers will help you trac4 down bugs or identify bugs within a specific build. 'or example someone might come to you with a bug on an earlier version. If you can identify what build they are wor4ing on and what build is completed, it could just be a simple fix of them getting the new build for the bug fix. It would be a good idea to send an email that tells people a new build is completed. #nother note on build is to ma4e sure to mar4 the build release or debug because there could be some problems with one but not the other. 7 Me*or) *a5ae*e5"
It is a good idea to chec4 the current status of the memory after a nightly build. If there are significant changes between the new day and day before, ma4e a note of it. !omething could be wrong. If the exe si1e grows too large then the game will ta4e longer to load. &7 De/0er a5% Co%e +5$8ec"+o5
#llot of your time is going to be spent on debugging or at least helping someone debug the game. :ne of the main s4ills a software tester should have is a !T%:? understanding on how a debugger 2isual (LL, Ainux ebugger6 wor4s so that they can accomplish this tas4. They should be able to step through or into parts of code and be able to understand what is happening and how to set and change variables as well as run a function at any time during the program. #t this time you will be doing some code inspection. The debugger can give you a lot of information on variables. se this information to verify that the code is using all of its generated nodes and arrays and that any discarded arrays or nodes are properly cleaned up so no null pointer or such are created. (hec4 the code for correct documentation for ease of use 2e.g. no fun4y variables li4e temp*6 ensure that the programmers are documenting correctly so their code can be pic4ed up easily in the off case someone will have to continue their wor4. =ou may have to also do some sort of data inspection depending on the game. If a game uses some sort of statistics program and randomi1ation, ma4e sure the data is being stored correctly
ame Testing 0ethodology and is only available to the parts of the program that use it. Ensure that the data is not getting corrupted as it is coveted and used in the pipelines. 7 Te$" Prora*$
# very important part of the job will be for a software tester to write test programs. !ome examples would be programs to enable cheat codes, play sound files, enable8disable the #I, recreate test scenarios, and whatever other debug information may be re$uired. The ame Tester also uses these test programs so they can test for specific gameplay scenario and8or situations. 67 So05% Te$"+5
!ound testing is simply listening to the sound and ma4ing sure the startup point and the length are appropriate. The software Tester often re$uires to write a &sound test) program to play all the individual sound files from inside the game 2just a $uic4 tip, you should chec4 with the sound artist and see if they have one already6. !ound testing includes testing if there will be any error in loading the files, listening to the sound files for errors or distortion. #lso, test to ma4e sure all the sound files are being loaded correctly. If the game has (olour (ommentary 2or ((6, use the cc profiler to analy1e. The analysis helps you to determine how many times a (( was called, and if it gets played or overwritten. The analy1er relates the scene or situations to the cc and chec4s to see if the information gets played, if you notice any problem in cc, you should chec4 if the information is being passed and played correctly. :f course one way to analy1e the cc is actually listening to it and note any irregularities and problems. <7 PD a5% 8+8el+5e$
It is important to generally understand how the pipelines wor4. +ow the sound is being processed and converted, how the 0aya art is being converted and used, generally how all the object data is exported from a raw data to a useable product. It would be a good idea to understand how P/ files are created and used. Aoo4 into the #nim classes, P/ chun4s and complex messes. =7 Da"a/a$e a5% Ga*e S"a"+$"+c$
The main purpose of the database for a game is7 6 Player or team statistics. It could be used solely for display, or it could b used by the #I engine to simulate the actual game play or for randomi1ation. 6 %un-time statistics. It is used by the game engine to remember certain state, attributes or scoring of the player as you move from level to level 2or to a different game state6. The easiest way to verify the database is to go into the game itself using a debugger, and analy1e the game is using the data correctly. :btain a printout of the database and loo4 at the screens to ma4e sure the data is being loaded, in the right place, showing the right information. :n a higher
ame Testing 0ethodology note, one could chec4 the database export to ma4e sure no error found in the export. #nother good techni$ue is to do random chec4ing, say, on a player statistic, to ma4e sure the data is correct. >7 O4erla)$
0a4e sure the overlays are loaded correctly. # tester could write a program to test that the overlays are being called correctly, and identify if any one does not get used at all. 1?7 Fro5" E5%
0ost of the 'ront End screens could relate to statistics 2you 4now, in the database6. The main thing to loo4 into is the &Text 3ible), i.e. the appropriateness of the text, the naming convention, chec4ing against any manufacture standards or re$uirements, and even spelling. 117 B0 Track+5 So;":are
The software tester should 4now how to use P(! Trac4er to trac4 and report bugs, and 4now the use of the standard template to create8customi1e a new defect database for a new project. It would be preferred if you could have administrator rights to trac4er to set up others with accounts and manage the database 2new client sign-on I, fields and information6. 127 Ga*e Te$"er a5% lo4e o; "#e a*e
# software tester must also have &to have a love of games) in mind. # significant amount of your time will be playing games to find the bugs. =ou have to have the s4ills to find and report game bugs as well as help repair them. It is important to test for the fun factor of a game as well, if you stop having fun while playing the game then ma4e sure to discuss with your peers li4e the ame Tester. If the changes happen too late they may not get implemented. !till, remember the debugger is your best friend. 17 B0r5+5 CD$
!imple, burn ( copies of the game for people who need them 2game testers, management6. # bigger part of this would be to trac4 down problems of bugs being discovered on a different medium.
ame Testing 0ethodology
<.
TESTING CONSIDERATIONS FOR SOFT(ARE TESTING
<.1
TESTING CO3ERAGE
It will be way too much for a !oftware Tester to handle if someone has to do <
<.2
TESTING 3S. DEBUGGING
5e all 4now good analysis of a bug simplifies the debugging process. +owever, there is a balance to stri4e between the amount of investigation and analysis done by a !oftware Tester and the amount done by a eveloper. +ere are some factors to consider7
<.
Isolate the problem so it can be assigned to a single person in the bug report. escribe the problem at the appropriate level of details so the developer can wor4 on it without wasting too much time to re-investigate and explore the problem. #re we in the &crunch) time F is the developer on the critical path> #re you a more s4illed debugger than the developer who is assigned for this bug> +ow we can get more productivity and better use of time>
TOOLS SUPPORT FOR TESTING
The ame Tester re$uires a fully functional Test Environment for testing. This test environment is comparable to a debugging environment that is used by the game developer or the !oftware Tester. #lso, the test environment must support testing the graphic components of the game 2e.g. providing some of the artwor4 run-time information* not just showing the artwor4 itself6. The !oftware Tester is expected to provide support and help out the ame Tester in using a new Test Environment platform.
ame Testing 0ethodology
=.
FULL CYCLE TESTING
ame development goes through a cycle of stages. The game software is defined, evaluated, designed, built, tested, fixed, #lpha, 3eta and then 'inal. The game development cycle involves many tas4s by multiple functional teams, and in multiple stages. 0ost of the stages are often described se$uentially but the underlying tas4s are often proceeding in parallel, especially during game production. Testing and fixing can happen at any stages in the project cycle. +owever, the cost of fixing an error increases exponentially as project progresses. The objective of &'ull (ycle Testing) is to identify and fix the error 2a4a defect or bug6 as early as possible so a $uality product will be delivered at the end. In a simple terminology, the techni$ues used for &'ull (ycle Testing) are 6 validation for accuracy and correctness, and 6 verification for completeness.
=.1
TESTING FOR DIFFERENT PRO@ECT STAGES
Re0+re*e5" $"ae game idea, storyboard, features and re$uirement document
Ga*e De$+5 $"ae conceptual design of the new features
=.2
Tec#5+cal De$+5 $"ae mainly, the Technical esign document and the 'eature !pecification. #lso, the game architecture, session manager, front-end, #I, director, rendering, animation, frame overlay and so on. This encompasses most of the components assigned to the team and support teams De4elo8*e5" $"ae coding and implementation8integration of various pipelines and wor4 by the teams that are integrated to ma4e up the game. &(lear box) testing will encompass module8component testing, testing of coverage and flow, data integrity, algorithm-specific testing 2e.g. debug code is in the game code6, path testing, and various levels of functional regression testing. This is more an incremental testing. Re4+e: $"ae 'ocus roup, and a &big bang) testing both (lear 3ox and 3lac4 3ox Testing, in a structured manner. Rere$$+o5 Te$"+5 $"ae it will be the game testing 2#lpha, 3eta, and 'inal6.
RE3IE( CRITERIA AND CHEC'LIST
+ere are the testing criteria for the re$uirement phase and the design phase7 %e$uirement ocument7 #re these the right re$uirements for the new game> #re they complete> #re they achievable> #re they realistic> #re they reasonable> #re they compatible to other competitor game> #re they testable>
ame Testing 0ethodology #re there $uantifiable and measurable targets> #re they in line with our direction of the company 2or company future6> ame and Technical esign ocument7 Is the design good> Is the design complete> Is the design doable> Is the design e$uipped with the necessary error handling> oes the design provide any debugging facility> Is the design in line with our game architecture> +as the design received review and approval>
ame Testing 0ethodology
>.
TEST PLAN AND TESTING DOCUMENTATION
Effective testing relies on a complete Test Plan pac4age. The document should be written in a way that promotes re0$a/+l+") 2reuse a large portion of the previous test plan documents for a new game6 and co5$+$"e5c) 2to minimi1e learning curve* the testing documentation produced for the game projects all have a similar format6. The hierarchical listing provided below is to outline a structure and the testing document framewor4 that the testing will be handled for each game project.
Testing %e$uirements ocument 2as explained earlier6 !oftware Testing Test Plan ame Testing Test Plan 2positive testing F what is expected>6 ame Testing !cenario 2negative testing F how robust is the game>6 0anufacture Platform Technical %e$uirements 2e.g. !:?= Technical %e$uirement (hec4list for the P!@8P! game titles, and 0icrosoft Technical %e$uirement for the @-3ox game titles6
The ame Testing Test Plan is a &positive testing) document that explains &'T and '& to test for the playability of the game 2e.g. options and features6. "# has developed a guideline to explain the !:?= re$uirements, section by section. 5e also reviewed the !:?= 3ug %eport from our previous game submissions and develop a “our lesson learned” chec4list so &e on*t !ake the sa!e !istake tice). #s far as timing goes, the review for the manufacture standards and technical re$uirements is often conducted after #lpha. In the Test esign document, you will learn the details 2e.g. definition, guideline, templates and ideas6 of preparing the test plan documents.
ame Testing 0ethodology
1?.
HO( TO CONDUCT EFFECTI3E TESTING
These are some of the &good ideas) from the experienced "# testers7
0a4e sure you are testing on the most up-to-dated version of the game. 5ith at least one version possibly created on the same day, it is easy to find yourself testing the wrong version. 0a4e sure that every time you get a new game build, you are getting every part of a new game 2a new game has more than one fileM6 and it may not wor4 against a previous saved game from a ame !ave8Aoad device li4e 0E0:%= card. Neep trac4 of your hours so you have time to loo4 for new bugs and regression test the &fixed) bugs that you need to verify for closure. # good understanding of the Trac4er database is a MUST. =ou must understand the Trac4er template, the definition and attributes of each field. #n optional field does not mean the field is not re+uired * the field is not re$uired o5l) +; it is not applicable to the bug. 5e must regression test the previous bugs. Hust because something got fixed in the previous versions is not a guarantee that it still wor4s in the current version. This is our mandate to retest the previously closed bugs with the severity of #-(rash, 3-?on functioning 'eature, and (- 0ajor 'unctional every wee4 and for each major milestone. %egression testing can also be frustrating, ma4e sure you stay focused and don9t thin4 of it as a trivial tas4. It is NOT only tester who can enter bugs to the defect database. +owever, you should review the bug report if that is entered by a non-tester person 2why> It is because you are responsible for regression testing, and the testers are responsible for all the bugs in the defect database6. Neep in mind that you will find bugs, however your productivity is not measured by the number of bugs that you find. There is no fixed rule for a game tester as the number of bugs that he or she must report in a day.
ame Testing 0ethodology
11.
ETERNAL TESTING
11.1 TESTING BY THE PUBLISHER 5hen a game is getting ready, we send the game to the publisher so they can ta4e a closer loo4 of the game and provide us with their feedbac4. %oughly to G wee4s before the 3eta milestone date, we start sending the Publisher with the game build twice a wee4 2e.g. every Tuesday and 'riday6. 5hen the game is 3eta, we will send the Publisher with our daily build of the game. 0ost of the publishers have their own 'TP 2'ile Transfer Protocol6 site where we can upload the game file over the Internet. =ou should find out the 'TP instructions from the Publisher. The Publisher sends us their bug reports wee4ly to almost daily as the game is getting closed to 'inal. =ou must be aware that the publisher tests the game remotely, and uses a build that is a few days older than our latest version. 5e often receive &crac4) bugs 2i.e. misinterpretation of the feature6 or bugs that have already been identified. This is the responsibility of the Test Aead to rectify these defect records before they are reviewed by the game designer or Producer. To facilitate the bug submission by the publisher, the Test Aead will provide with a file exchange format and the P(! Trac4er efect %eporting template to the publisher so their bug data can be imported to our defect database.
11.2 TESTING BY THE MANUFACTURER 5hen the game is 2or closed to6 'inal, the game is sent to the game platform manufacturer for testing. The primary focus of testing is on the peripherals and devices 2e.g. controller and game pad6, the naming convention, standards and the legal text appeared in the ?I! or menus. The tester must be familiar with the Technical %e$uirements for each of the game platform. sually there are two 4inds of submission to the manufacturer7
Pre-$0/*+$$+o59 The game is sent to the manufacturer 2e.g. !:?=, 0icrosoft6 so they will briefly loo4 at it and provide us with some feedbac4. Pre-submission gets us some idea about the readiness of the game. This is sort of an informal review, if we don9t receive any problem, it doesn9t necessarily mean the game is bug free. 5e often have a couple of pre-submissions for a brand new game 2e.g. our first action game of Hac4ie (han !tuntmaster6, and we may s4ip the pre-submission for the second generation of a previous game title. S0/*+$$+o59 The game is formally reviewed by the manufacturer, usually a team of people is assigned to test the game. The result in the form of a manufacturer bug report is sent to us through the publisher. The submission process ta4es about a wee4. # game must meet certain standards and re$uirements 2e.g. no crash bugs, no more than certain number of noncritical bugs identified by the manufacturer6 before it is considered “co!!ercial read#” by the manufacturer, and no game can be published without manufacturer9s approval.
There are certain rules that must be followed for the submission procedure, e.g. the use of special ( media, the number of copies, and the game pac4aging, etc. Please refer to the manufacturer specification for details. The Test Aead must wor4 closely with the Producer, the Publisher and the testing group of the manufacturer for the submission process.
