T R A I N I N G EV E V A L U A T I O N P R O CE CE SS SS NHS CFH
National Training /Bowo Evaluation Process Witoyo/
Training Evaluation Process Evaluation should not be confused with assessment as they are sometimes thought of as the same, but there are clear differences between the two. Assessment is an ongoing process aimed at improving delegates’ learning, which is undertaken: ·
before (pre-course);
·
during the course (formative); or
·
a final (summative) quiz or test based around the learning objectives.
The outcomes outcomes of these processes need to be fair, reliable and valid. This data can then be used to evaluate the learning and enable an accurate measure of the intervention at a given time. Evaluation therefore is typically a broader concept than assessment as it deals with all aspects of a programme including resources, staffing, organisation, operations, efficiency and the overall overall measurement measurement of the learning that has been been delivered. It can therefore be described as the systematic acquisition and assessment of information to provide useful feedback about some object, in this case training.
The Evaluation & Learning Life Cycle The evaluation and learning cycle shows how evaluation fits within the overall context of any learning and development programme.
Evaluate Business Needs
Evaluation Feedback
Training Needs Analysis Design Learning Solution NHS NPfIT
Validate Transfer Reinforce Learning
Training Delivery
Validate Learning
Evaluation is an integral part of the learning cycle and should not be considered in isolation.
© Crown 2007
Page 6 of 13
NHS CFH
National Training Evaluation Process
All education, training and development programmes should begin with a training strategy that clearly links the business objectives and benefits to the training objectives, so that the training needs analysis (TNA) can be used to identify the real training needs of the target audience. The TNA will define the training curriculum and high level content of the proposed training programme. programme. Alongside this activity the evaluation evaluation and assessment assessment methodology will be defined detailing the evaluation objectives, the data collection process and the analysis analysis and reporting mechanisms. mechanisms. The evaluation questions must must be linked directly to the training objectives for the evaluation to be meaningful. The actual evaluation phase of the training programme will then be broken down into several distinct phases: Validation, Reinforcement, Transfer of Learning and Evaluation Feedback.
Questionnaire Design The on-line questionnaires should be spilt into several sections containing the following subject areas: Generic questions relating to the following: ·
Trainers performance
·
Documentation quality
·
If the course met the objectives
·
If the course met the delegates needs
·
Training room facilities.
·
Course logistics (joining instructions, car parking etc)
Application and course questions: ·
Questions related to the specific learning objectives of the training course.
These questions are very important as this will enable the measurement of the learning achieved on the training course and if that learning has been transferred into the work place. It is these questions questions that will give the individual Trusts the ability ability to evaluate their own training courses. A number of evaluation templates are included at the end of this document.
Evaluation and Accreditation In addition to the evaluation of the learning objectives it is proposed that testing and accreditation is introduced into both the Train the Trainers (TtT) courses and the enduser training programmes. Note: Although currently out-of-scope of this project, we should encourage that these assessments be fed back to the delegate’s personal development plan (PDP) and the Knowledge and Skills Skills Framework (KSF). On a much wider scale each job job
© Crown 2007
Page 7 of 13
NHS CFH
National Training Evaluation Process
description should include IT competences that are relevant to the systems used by the post holder. To implement this would require a separate project in its own right, however NHS CFH should recommend that this be dealt with at a local level by the individual SHA’s and Trusts.
Learning Management System With the emergence of elearning and Learning Management Systems (LMSs) there is an impact on training evaluation. Although the overall philosophy remains remains the same, due to the very nature of elearning programmes, the evaluation process is made much simpler with feedback data being captured from tests and practical exercises readily available for online analysis and reporting. However this proposed process includes face to face classroom training as well as elearning so that all evaluation questionnaires and tests are delivered electronically so that the data can be captured for later evaluation and review. Training Evaluation Administration To enable the successful implementation of the national training evaluation process it will be necessary for the NHS training organisation in each Trust or Shared Service to recruit at least one training administrator to manage the training course evaluations. The duties duties will include the: ·
The on-line creation of questionnaires and tests.
·
The production of on-line reports and evaluations
Analysis & Reporting The system will support the: ·
Creation of tests that track and score learner performance either as topics within a course, or as standalone assessments.
·
Enable the review of test results and analysis of trends through test statistics.
·
Manage test question banks to enable extensive re-use of questions.
·
·
·
Optionally restrict repeated attempts of a test, and set a time delay between attempts. Define which questions are mandatory or optional. Creation of a suite of management reports that can be requested on an adhoc basis by the training administrators.
Capturing data from the questionnaires and tests will provide the foundation to empower Trusts to evaluate their own training, whilst at the same time enabling if the need arises the evaluation of pan SHA, LSP, NASP training programmes etc.
© Crown 2007
Page 8 of 13
NHS CFH
National Training Evaluation Process
System Access The training administrators will require access to be able to create and maintain a sub-set of the above questionnaires, tests and reporting requirements for their specific Trust. End users will require access to the questionnaires and tests via the Internet/Intranet to complete the questionnaires and tests on-line. It is envisaged that Trust administrators will only be able to access their own data to create questionnaires and analyse, however there is a requirement for evaluation across the NPfIT nationally. nationally. Therefore access access is required to enable enable a national training administrator to look at all Trusts data. Administrator Training Each training administrator will require systems training in the functions of the National LMS, questionnaire creation and maintenance and the reporting capabilities of the system.
© Crown 2007
Page 9 of 13
NHS CFH
National Training Evaluation Process
Training Evaluation Process Training Evaluation Process will consist of a number of stages which can be represented by the following diagram:
Level 1 to 3 Evaluation
Level 3 Delegates Post-Course Interview/Questionnaire
Line Managers Post-Course De-Briefing Level 2 Questionnaire & Test Level 1 End of Course Questionnaire Line Managers Pre-Course Briefing Delegates Pre-Course Assessment
© Crown Copyright 7
Provides evidence of the transfer of learning into the workplace
Provides the opportunity for reflective learning & further discussion about L&D opportunities
Provides evidence of the learning achieved and can be used for the accreditation process
Provides evidence of how the delegate has reacted to the learning
Positions the L&D intervention in the correct context – context –
Provides baseline measurement and assessment (TNA) of pre-requisite skills
Page 10 of 13
NHS CFH
National Training Evaluation Process
Level 4 and 5 Evaluation
Level 5 – Return on Investment
Provides evidence of the ROI by identifying the benefits achieved versus the cost of the training.
Measure ROI by reviewing:
Can be identified by using the formulae:
•Cost of Training •Benefits of the Training
ROI % = Benefits of Training (£) X 100 Cost of Training
Level 4 – Impact on the Organisation Analyse organisational performance by:
•Observation •1:1 Interviews •Measurement of quantifiable process
Provides evidence of the impact of the learning on the organisation.
identified in business analysis stage.
Level 4 and 5 evaluation will not be covered in the first stage of the implementation until levels 1, 2 and 3 have been successfully implemented.
© Crown Copyright 7
Page 11 of 13
NHS CFH
National Training Evaluation Process
Evaluation Framework The “Best Practice” evaluation process shown above forms the framework for the proposed National Evaluation Process which follows the principles of Kirkpatrick’s (level 1 to 4) and Phillips (level 5) models of evaluation. The fundamental principles of training evaluation described in these models are: ·
Level 1 - Shows the delegates reaction to the learning
·
Level 2 - Testing the learning achieved
·
Level 3 - Transfer of learning to the workplace
·
Level 4 - Results, or impact of the training on the organisation
·
Level 5 - Return on Investment
However for this evaluation method to fully complete the cycle it is necessary to include an earlier step of pre-training measurement and assessment of pre-requisite skills. In principle, it should be possible to evaluate the Return on Investment (ROI) from training (Level 5 above), with a calculation that measures cost reductions or improvement of results in financial terms, and demonstrates by what factor or multiple outweigh the costs of the training itself. In practice, however, very few organisations are able to specify the clear business benefits that derive from training, and even fewer are able to specify specify these business benefits in financial financial terms. In addition, the ROI will always be affected by a number of unknown variables, e.g. changing business priorities. As it will take some time to implement the full evaluation cycle it should be implemented step by step, with level 4 and 5 only being attempted when the organisation has fully accepted the principles of training evaluation and implemented the core process. The measurement of the ROI will be achieved at a later date in conjunction with the evaluation of benefits realisation.
© Crown Copyright 7
Page 12 of 13
NHS CFH
National Training Evaluation Process
Evaluation Building Blocks Capturing relevant training course data based on learning objectives, which are linked to the business benefits, are the key building blocks of any training evaluation process. This can be represented represented with the following following diagram: diagram:
Analysis
Design
Delivery
Post Training Evaluation
Evaluation Analysis
Training Programme Evaluation
Business Objectives
Training Materials Materials
Level 1 & 2 Questionnaires & Tests
Level 3 Questionnaires
NASP/LSP Evaluation
Pan SHA/Trusts Evaluation
Training Objectives
Evaluation Questionnaires
LMS Database
In this diagram the business objectives are identified at the earliest opportunity, and through the training needs analysis (TNA) process the training objectives are developed. It is fundamental to the success of any training evaluation evaluation exercise that the evaluation questions are linked to the training objectives and then the business objectives as shown in the analysis and design phases of the above diagram. The level 1 questionnaire is then used in the training delivery phase to capture feedback from the delegates at the end end of the training course. course. The level 2 questionnaire is used to formally test and record the delegates’ knowledge with questions constructed around the training objectives. objectives. Likewise the level level 3 evaluation documents are used to measure the impact of the learning some weeks or even months after the training programme programme has been completed. completed. In all cases the data captured from these documents and tests must be stored in a database for later analysis Once these fundamental principles are in place and data is being captured at the level of the individual training course, it will be possible to analyse the effectiveness of the training both at local and national level. In addition the recording recording of this data will enable large-scale evaluation, e.g. pan Trust, SHA, LSP and NASP training activities. Failure to capture the data at the level of the individual training course will negate the whole process and make overall evaluation and review of any training programme – much less comparisons between programmes, impossible.
© Crown Copyright 7
Page 13 of 13
