Nlng,Ats Shl

The Technical Test Battery Manual & User’s Guide INTRODUCTION TO THE TTB Rationale Short Descriptions of the Tests Choice of Tests Administration Scoring Technical Support Distribution Controls

RATIONALE The use of ability tests in the technical and engineering sectors of the economy is well established. Since its launch in 1979, SHL’s Technical Test Battery (TTB) has been widely used for the assessment and selection of engineering and craft apprentices as well as technicians and process operators. The emphasis in the TTB has always been on the assessment of relevant work-related skills rather than the measurement of abstract conceptions of human intelligence. Accordingly, the TTB has been updated so as to reflect changes that have taken place in the workforce since its original launch. Following consultations with major users and extensive re-analysis (including sex and ethnic bias), we have replaced or modified several of the original tests. The new tests are an improvement in four main respects: 1.

They are more up to date

2.

More reliable

3.

More valid

4.

Less prone to bias.

Each of the eight tests in the new edition of the TTB is presented separately to ensure greater flexibility, enabling users to select the most appropriate mix of abilities to be tested for a given job. Additionally, two versions of the most popular tests will be available to allow use of parallel versions on different occasions. This is valuable in helping to avoid any potential problem which may result from the over-exposure of individual tests, as well as in the retesting of individual candidates.

© SHL GROUP plc, 2000

As previously mentioned, the TTB can be used for the assessment and selection of staff for a wide range of technical jobs. The types of industry for which the TTB has been used include: paper and printing, gas, railways, heavy engineering, glass, the motor industry, chemicals and pharmaceuticals, the building industry and metal refining and processing. This list is by no means exhaustive of the areas in which the tests have been or could be used. The list should also be considered to include jobs in both the public and private sectors.

SHORT DESCRIPTIONS OF THE TESTS The updated TTB is composed of eight tests, and these are summarised below. Parallel versions of four tests, namely, NT2.2, MT4.2, VT5.2 and ST8.2 will be available a few months after the initial launch of the updated TTB. Verbal Comprehension (VT1.1) – measures vocabulary and basic word skills using language which reflects the requirements of technical occupations. Numerical Computation (NT2.1 and NT2.2) – assesses basic number skills with the emphasis on straightforward calculations and the understanding of arithmetical operations. Visual Estimation (ET3.1) – measures spatial perception involving the comparison of shapes and lengths. Mechanical Comprehension (MT4.1 and MT4.2) – measures understanding of basic mechanical principles. Technical Understanding (VT5.1 and VT5.2) – measures a candidate’s understanding of written passages containing the type of verbal material likely to be encountered in a technical setting, such as equipment manuals and operating instructions. Numerical Reasoning (NT6.1) – measures more advanced number skills, with the emphasis on reasoning and recognising short cuts to reach solutions, rather than complex calculations. This test has been developed specifically for use with hand-held calculators. Fault Diagnosis (FT7.1) – measures the candidate’s ability at finding faults in systems. The candidate must locate which element of an arrangement of processors is not working as specified. Spatial Recognition (ST8.1 and ST8.2) – measures the ability to recognise shapes in two dimensions, even after rotation.

CHOICE OF TESTS Choice of tests should be founded on careful job analysis, which will identify the abilities and attributes required to perform the job. The resulting, well researched person specification, read in conjunction with test descriptions, validity evidence, etc, presented in this manual and elsewhere, should enable the selection of an appropriate combination of TTB tests. Wherever possible, validity research should be conducted prior to the introduction of tests to ensure that they are appropriate for their intended purpose and fair to all applicants.


2

ADMINISTRATION The administration of the TTB tests is a straightforward procedure through the use of separate administration cards. The total testing time for this battery is 1 hour and 50 minutes. With administration and example questions, ti would take just over 3 hours for all the tests to be completed. These tests can, however, be used singly or in various combinations to establish a profile of an individual’s abilities depending on the skills required for a particular job. Experience has shown that a selection of three or four of the tests is typically more appropriate, and most combinations of this number of tests can be completed in about 1½ hours.

SCORING The tests in the TTB can be scored either by hand or machine. They are all in multiplechoice, paper and pencil format. The multiple-choice format and special answer sheets permit quick scoring using plastic scoring key overlays. In additions, the tests can be scored ® using the NCS “Opscan 5 ” optical mark reader, connected to a personal computer. Machine scoring is particularly effective where a large number of applicants need to be tested. (Please contact SHL’s Computer Services Department for further information.)

TECHNICAL SUPPORT The Technical Test Battery is supported by SHL’s ongoing research and development programme and test users are regularly updated with recent norms and validation studies from a range of organisations. Norms for particular organisations can be provided on receipt of raw scores (and basic biodata such as age, sex, educational qualifications), and test users are encouraged to send data to us. Advice or assistance with job analysis and validation studies is also available. Such research is important to ensure test relevance as only relevant measures contribute to the effectiveness and intrinsic fairness of selection decisions.

DISTRIBUTION CONTROLS These tests are not freely available to the general public, but are distributed under strict control to suitably trained and qualified test users. SHL runs training courses in occupational testing to standards laid down by international psychological associations.


3

2.

DESCRIPTION OF THE TTB TESTS

Overview Verbal Comprehension Numerical Computation Visual Estimation Mechanical Comprehension Technical Understanding Numerical Reasoning Fault Diagnosis Spatial Recognition

OVERVIEW The SHL Technical Test Battery is a collection of tests designed specifically for the selection and allocation of a wide range of apprentice and technical personnel. The eight tests in the TTB can be divided into two groups. NT2.1, MT4.1, VT5.1 and ST8.1 made up a group of core skills, while the remaining tests measure more specific aptitudes. Together, the tests provide for a wide ability range from those with no formal educational qualifications through to GCSE or ‘O’ level and ‘A’ level standard. All the tests are available in individual booklets and are designed to be used separately or in any combination. The choice of test or tests to be used should be based on the results of a job analysis, a detailed job description or a validation study. This will identify the attributes required to perform effectively in the job.


4

The time limits and number of items in each test are given below: TEST

NUMBER OF ITEMS

TIME LIMIT (minutes)

APPROX. ADMIN TIME (minutes)

VT1.1

Verbal Comprehension

36

12

20

NT2.1

Numerical Computation

36

10

20

ET3.1

Visual Estimation

36

10

20

MT4.1

Mechanical Comprehension

36

18

30

VT5.1

Technical Understanding

36

20

30

NT6.1

Numerical Reasoning

25

12

20

FT7.1

Fault Diagnosis

36

18

30

ST8.1

Spatial Recognition

36

10

20

More detailed descriptions of the individual TTB tests follow. In order to provide a clear understanding of the item format, copies of the example pages are also included. These reflect only the easier items. For an overview of the complete test, the question booklet itself should be consulted. The indications given in this manual of the jobs for which these tests may be applicable are intended as guidelines only; test relevance should be examined in relation to each individual job. Users who have concerns about test choice should seek professional advice before implementing the tests. SHL does not accept liability for any loss of whatsoever nature suffered by any person or entity as a result of placing reliance on this Manual and User’s Guide.

VERBAL COMPREHENSION

VT1.1

The TTB Verbal Comprehension test (VT1.1) has been designed as a measure of basic word skills suitable for a wide range of occupations, from basic craft apprentices up to those of technician or supervisory level. The item format of VT1.1 is varied and includes sentence completion, analogies and synonyms. The language, which has been selected to reflect the requirements of technical training, is relevant to most semi -skilled jobs within the technical sector; however, no prior knowledge of ‘technical’ words is assumed. Indeed, even though many of the sentence completion items are set in a technical context, no specialist knowledge is required to select the missing word. Some of the language used for other questions in this test is deliberately of a more general nature, since, in any work environment, a reasonably wide vocabulary is essential if an individual is to communicate effectively. In addition, the test is not highly speeded, with the majority of candidates being expected to complete the test within the time allowed. Time limit:

12 minutes

Number of items:

36


5

Range:

GCSE level up to ‘A’ level.

Total administration time (including instructions) approximately 20 minutes


6

NUMERICAL COMPUTATION

NT2.1 and NT2.2

The Numerical Computation tests have been designed to measure the four basic arithmetic operations of addition, subtraction, multiplication and division. In most modern technical environments, calculators are frequently used to perform basic calculations, so saving time and reducing the number of computational errors. This tendency © SHL GROUP plc, 2000

7

to use calculators at every opportunity, however, increases people’s dependence upon them and reduces their ability to spot any errors which may occur. The emphasis in this test is placed on the understanding of the relationships between numbers rather than straightforward computational ability. The applicant is required not only to calculate but also to follow and choose the correct sequence of mathematical operations in order to derive the correct answer: skills which remain of crucial importance even in the age of the calculator. Candidates are required to work through problems where one operation or number has been omitted from a sequence and to select the missing term from five alternatives. Percentages, decimals and fractions are also covered in this test but there is a deliberate avoidance of the more advanced number-skills which are covered in the Numerical Reasoning test (NT6.1). Time limit:

10 minutes

Number of items: Range:

36

No formal qualifications up to ‘A’ level.

Total administration time (including instructions) approximately 20 minutes.


8

VISUAL ESTIMATION

ET3.1

The Visual Estimation test has been designed to measure a candidate’s ability to accurately estimate length and breadth as well as to identify minor differences in a variety of basic shapes. Some examples of activities where visual estimation skills are useful are: a fitter checking parts prior to assembly, a turner machining a precision component, a joiner making a close fitting joint and a heating engineer estimating pipe runs. In this test, the candidate is asked to choose the two figures from a set of five which are identical. The test is arranged so that progressively finer discriminations are required in order to select the two identical figures. These differences range from 3 mm in the early items to 1 mm towards the end of the test. Time limit:

10 minutes


36

Suitable for any educational level.



9


10

MECHANICAL COMPREHENSION

MT4.1 and MT4.2

Unlike many previous tests of mechanical comprehension which purport to measure an individual’s understanding of basic mechanical principles, the emphasis here is on more spatially orientated items, which require the candidate to visualise movement, rather than recall situations obviously based on ‘textbook’ laws of physics. An effort has been made to select items aimed at identifying aptitude and potential rather than current knowledge which may be better measured by other means, eg GCSE results. The test is made up of 36 questions, each relating to a three-dimensional drawing. Most items involve simple mechanical devices, such as gears, pulleys, levers and lifting mechanisms. Due to different levels of interest and involvement between men and women and between people of different ethnic backgrounds, mechanical comprehension is an area where group differences are likely to occur. Therefore, when employing these tests, the user should always ensure that these skills are relevant to the job or function in question. Time limit:

18 minutes


36




11

TECHNICAL UNDERSTANDING

VT5.1 and VT5.2

The Technical Understanding tests measure the ability to follow written instructions in a technical context. Candidates are presented with passages of written information. After reading each passage, they have to answer four multiple choice questions which test their understanding. To use new equipment and appliances properly often requires an operator to read, and follow precisely, written instructions laid out in detailed manuals. Even the most basic of written instructions are capable of misinterpretation and, apart from avoidable delays, mis-operation of equipment could lead to its damage or compromise the safety of the operator or others. The topics covered include operating instructions and procedures associated with a wide range of equipment and technology, just the kind of materials to be found in technical manuals or system guides. No prior knowledge of technical words or jargon is assumed and so applicants with little or no technical background can be assessed.


12

This ability is relevant to a wide range of technical work, particularly that involving written documentation for the installation and maintenance of machinery and also in the training of technical personnel. Research into VT5.1 has shown group differences which appear to reflect the amount of verbal information contained in the test. This is particularly true for individuals for whom English is not the first language. Users must therefore ensure that the verbal loading introduced by this test is justifiable against job demands. See section 10, ‘Testing and Equal Opportunity’ for further details. Time limit:

20 minutes


36

From no formal qualifications through to GCSE or ‘O’ level.



13


14

NUMERICAL REASONING

NT6.1

In this test the emphasis has been placed on reasoning skills rather than on the simpler number calculations covered in the Numerical Computation tests (NT2.1 & NT2.2). The Numerical Reasoning test comprises written problems, expressed within the context of technical work, which involve percentages, fractions, decimals and diagrams, as well as knowledge of the four basic rules of arithmetic. The need for long and tedious calculations has been avoided in NT6.1. It is expected that candidates will spend more time in generating and choosing strategies to solve numerical problems rather than in calculating their answers. Candidates are allowed to use calculators in this test to reflect the wide availability of such aids at virtually every workplace. The test has also been designed to give better discrimination at high levels compared to the Numerical Computation tests NT2.1 and NT2.2. Nevertheless, for those with still higher numerical skills, such as graduates, the Numerical Critical Reasoning tests from the MGIB and CRTB will be more suitable. Time limit:

12 minutes


25

Suitable for GCSE and GCE ‘O’ level groups up to GCE ‘A’ level candidates.



15

FAULT DIAGNOSIS

FT7.1

This test is designed to measure candidates’ ability to locate faults in systems. The growing complexity of equipment, especially that associated with electronics and computing, brings with it greater demands in terms of maintenance and servicing. The ability of an individual to isolate faults and problems in a system will therefore be of increasing importance in reducing maintenance costs and equipment downtime. This test measures the ability to locate faults in a logical system. The test requires an individual to locate which element in an arrangement of coded symbols is not working as specified. The test is based upon a PROCESS CONTROL PANEL, which contains ten buttons or processors. The control panel, together with a complete listing of all processors and their functions, is shown on each double page. No specialised knowledge of fault-finding or electronics is required and so applicants with little or no technical background can be assessed. The ability to find and locate faults is relevant in a wide range of areas both in machine maintenance and electronics, and also in software, process control systems and communications.


16

This test could be compared with FTS4 (from the Applied Technology Series) but, since it is of a slightly lower level, it is suitable for candidates with little educational attainment up to ‘A’ level standard. The test could also be used for candidates with some work experience. Time limit:

18 minutes


36

From GCSE upwards to ‘A’ level.



17


18


19


20

SPATIAL RECOGNITION

ST8.1 and ST8.2

These tests measure the ability to recognise and match shapes in various orientations. This skill is of crucial importance in this design, manufacture and assembly of individual components and machinery. The candidate is asked to choose the one figure from a set of five which is identical to a given shape. The choices are often rotated relative to the given pattern and the test is organised so that progressively finer discriminations are required in order to select the identical figure. This ability is likely to feature in the design, production and maintenance of machine tools, engineering components, mechanical sub-assemblies and other activities involving designing, fitting and inspection. The level of spatial aptitude measured by this test is probably slightly lower than that measured by STS5 (from the Applied Technology Series) or SIT7 (from the Information Technology Test Series). Time limit:

10 minutes


36




21


22


23

3.

THE DEVELOPMENT OF THE TTB

Background Defining the Concept Test and Item Design Test and Item Review Item Trialling Item Analysis Final Form Trials and Analysis Manual Writing and Ongoing Technical Reports

BACKGROUND Like all SHL test batteries, the TTB has been through a lengthy development process. The original TTB tests were first developed in 1979. A full review of all the TTB materials was then carried out in 1991. Although some of the present test content is based on the 1979 work, this has all been revised, amended and supplemented by new material in addition, all the tests have been retrialled. The most relevant description of the development process for the present battery is therefore this current revision rather than the original development, and it is this which is mainly reported here.

DEFINING THE CONCEPT The original TTB was developed after extensive research in the technical/engineering sector. Several key skills, required by those employed in this sector, were identified. These skills were reviewed and refined in 1991 on the basis of further research, recent validation studies and consultation with many companies and training organisations with an interest in this area. Broadly speaking, the skills which were found to be important twelve years ago are still relevant today.

TEST AND ITEM DESIGN The rationale for each of the TTB tests, including information on the sources used for test content, is summarised below.


24

Verbal Comprehension Designed to measure basic work skills, Verbal Comprehension is a vocabulary based test which uses a variety of item types to measure a candidate’s understanding or application of words. The language used has been chosen to reflect that which may be encountered in a ‘technical’ working environment although obscure, out-dated, and jargonistic words have been avoided. The test has generous time limits which allows most candidates to complete it within the time allowed. Numerical Computation These tests measure the ability to identify and carry out standard arithmetical operations without the aid of a calculator. Although it is recognised that aids to calculation are readily available in most work places, there are still situations where dependence on them for simple computation could lead to significant (and often unacceptable) reductions in efficiency. The test content has been made context -free to ensure applicability across a wide range of jobs. Most items contain operations on whole numbers, but decimals, fractions and percentages are also included in the test. The format of the items requires candidates to demonstrate an understanding of the relationships between numbers and arithmetical operations rather than relying purely on calculation alone. This results in a measure which samples a more complete set of basic number skills. Visual Estimation The ability to make accurate visual comparisons between various components is an essential part of many manufacturing jobs, as is the ability to gauge size without the constant need to take repeated measurements. Visual Estima tion is specifically designed to measure this important ability. The original test format has been completely revised, with uniform rectangles replacing lines and items involving angles being dropped. The test gets progressively more difficult and requires candidates to make finer and finer discriminations. Visual Estimation is a straightforward test to administer and is suitable for groups of any educational level. Mechanical Comprehension This test is designed to measure the ability of candidates to understand the workings of simple mechanical devices by recognising and applying appropriate mechanical principles. The aim of Mechanical Comprehension is to identify aptitude rather than current knowledge as this may be better measured by academic examinations. Items obviously based on ‘text book’ laws of physics have, as far as possible, been avoided with many items requiring candidates to visualise movement in order to find the solution to a problem. Whilst every effort has been made to choose items which are fair and equally predictive for all groups, our experience of this test is that males are generally more likely to score higher than females. This difference is usually smaller when males and females from actual applicant groups are compared. However, it is important that users of this test are able to justify its use in relation to the requirements of the job, ie, is it relevant? does it measure a key skill in the job?


25

Technical Understanding The ability to read, understand and act on written information is a fundamental skill in the technical workplace. Technical Understanding is designed to measure this ability using passages of text drawn from a variety of sources including technical manuals, operating instructions, maintenance schedules and fault finding routines, etc. Each passage is followed by four questions aimed at testing the candidates’ understanding. Although there are generous time limits for this test, the amount of verbal material involved may lead to adverse impact against some ethnic minority groups. Users need to ensure, therefore, that the test is directly relevant to the job requirements before using it. Numerical Reasoning This test is designed to measure candidates’ reasoning ability when handling numerical information. The data is laid out in the form of short written problems, some of which are supported by diagrams. The candidate must first decide how to solve the problem and then correctly apply the relevant arithmetical operations to complete it. Since the emphasis here is on understanding and reasoning with numerical information, this test has been developed specifically for use with a hand-held calculator. This reduces the computational load of the test and ensures that those candidates who have adequate reasoning ability but not always the computation skills to match, are not penalised unfairly. Fault Diagnosis This test measures the ability to locate faults in a complex but logical system. This diagnostic ability has become increasingly important in recent years with the rapid rise in new technology. Individuals who are able to isolate faults and take the necessary remedial action are likely to reduce maintenance costs and equipment downtime. The test requires candidates to identify, by working back from a given output, the one processor in a sequence which is not working as specified. Fault Diagnosis is a task orientated test unlike the more abstract Diagrammatic Reasoning (DT8) from the original TTB, which it replaces. As such, it is more directly relevant to the requirements of modern industry and is likely to have increased validity. Spatial Recognition This test is designed to measure the ability of candidates to recognise complex shapes (components) even after they have been rotated. This is an essential skill in many areas of manufacturing and production and is relevant to a wide range of occupations including machinists, fitters, motor mechanics and service engineers as well as draughtspeople and designers. The item format is based on a selection of shapes which have been chosen to reflect modern component design. Candidates are given a shape on the left hand page and are asked to choose from five alternatives on the right, the one which is identical. Items toward the end of


26

the test get progressively more difficult, as the differences between the given shape and five alternatives become less obvious.

TEST AND ITEM REVIEW As mentioned earlier, the original TTB tests were developed and trialled in 1978/9. In the 1991 revision, all of this original material was thoroughly reviewed and sufficient additional material written to enable parallel forms to be constructed. (The provision of sufficient extra items enabled those not performing to acceptable standards to be rejected at a later date.) All of the material was then further reviewed and amended by a team of experienced psychometrists within SHL, at two levels. First, the overall design of each test was considered in relation to its credibility(ie, face validity) and job relevance (ie, content validity). Secondly, within each test, every item was reviewed individually in order to eliminate those which did not appear to be rationally related to other test content, those that were ambiguous in any manner or those that were judged to be in any other way unsuitable. Finally, the material was reviewed by equal opportunities specialists and members of minority groups to eliminate any items which could be identified as having a potential to create gender or ethnic bias.

ITEM TRIALLING All of the items which met the reviewing criteria were assembled into trial test versions in an order which was a close as possible to the intended final forms. Each test was trialled on a sample of around 200 people drawn from various companies, applicant groups and final year school and further education students; a sample intended to be as representative as possible of those on whom the tests might be used. In total, some 800 people took part in these trials which were held between March and July 1991.

ITEM ANALYSIS The next stage was item analysis. Here statistical procedures were used to investigate the pattern of responses to each item in detail. Each item was considered individually and in relation to the rest of the test from which it was drawn. Items were rejected if there was inconsistency in response patterns, if they did not relate to the other items in the test or to the construct being measured, or if there appeared to be a difficulty or ambiguity which resulted in otherwise high scoring trial participants responding incorrectly. The response patterns of different ethnic and gender groups were also investigated at this stage using methods such as the Mantel-Haenszel procedure. Items were rejected where they were found to show differential difficulty for a particular group unrelated to the ability being measured by the test. (Full detail of the results of the item analyses are shown in Section 12, “Additional Technical Data”.)


27

FINAL FORM TRIALS AND ANALYSIS The remaining items meeting the item analysis criteria were then selected for inclusion in the final forms of the tests. In choosing these items a number of considerations were addressed. The items which were selected had to reflect the full range of content of each test and provide a span of difficulty levels from very easy items through to those which the trial groups found most difficult. Within these constraints, the items with the best psychometric properties were chosen. The final form of each test was re-trialled during the period October 1991-April 1992. Additional groups were used in order to confirm that the tests performed psychometrically as anticipated and to provide standardisation (norm) data. A further aim of these trials was to collect calibration data to allow scores on earlier versions of each test to be directly compared with those from new and updated versions. This ensured that norm data collected on one form of the test could be converted for use with other forms. For this purpose, groups of trial subjects were asked to complete more than one form of a test, wherever possible. Standardisation data was collected from a sample made up principally of apprentices in and apprentice applicants to the engineering industry (76%). Their educational level varied from no formal qualifications up to ONC level. The remainder (24%) of the sample was made up of final year students (studying a range of subjects) and work experienced adults in a variety of disciplines within the technical field. In terms of ethnic origin, 87% were classified as white European and 13% belonged to a combined ethnic minority group (Asian, African, Afro-Caribbean – according to selfdescription). In terms of gender, 89% of the group were male and 11% were female. Altogether 1 937 people took part in these trials.

MANUAL WRITING AND ONGOING TECHNICAL REPORTS The first (1979) edition of the TTB manual contained detailed information on administration, scoring and norming as well as technical information on all eight tests. Since original publication, much more information on the tests has been collected through technical studies and follow up on the tests in use. In 1982, a Normline supplement was published containing norm data for a variety of groups and details of test validation. In addition, a further spatial test (ST9) was added to the battery. Later, a practice leaflet was produced to allow users to provide candidates with information about the tests in advance and some sample questions. In 1987 and 1991, further Normline Updates were published and made available to holders of the original TTB manual. This present (1992) manual contains original material and incorporates new norm data, details a new validation studies and an expanded discussion of Testing and Equal Opportunity. Monitoring the TTB is an ongoing process and we will continue to publish new information from time to time. Test users are urged to contribute to the process by supplying SHL with scores and details of their own findings with the tests which can then be shared with other TTB users through the publication of appropriate norms and technical updates.


28

Nlng,Ats Shl

Recommend Documents