DETERMINING RETURN ON INVESTMENT IN TRAINING/EDUCATION Introduction Quantitative Assessments Benefit-Cost Ratio (BCR) Examples Worksheet
Return on Investment (ROI) Examples Worksheet
Benefit Forecasting Examples Performance Values Worksheet Costs Worksheet Worksheet
Cost Effectiveness Example Worksheet
Qualitative Assessments Data Collection Qualitative vs Quantitative Example Implemention Process Evaluation When to Use
Mixed Methods Advantages Examples Worksheets
References
1
Introduction With companies scrutinizing scrutinizing costs and tightening purse strings, how do they prove that training/education is worth the investment? A first rate evaluation study may be necessary to justify the investment of time and capital in a training/education training/education offering. The classic model to evaluate training/education offerings offerings developed by Kirkpatrick (1998) looks at four levels of assessment: •
Student reaction,
•
Learning,
•
Behavior, and
•
Business results. When looking at the business results of training, training, many critics say the Kirkpatrick model
does not go far enough in analyzing the impact of training/education on a company’s bottom line. Training/education offerings should be evaluated at a fifth level-level--financial financial return. return. There are several methods for evaluating training/education offerings based on financial return and these methods include both quantitative and qualitative assessments.
Quantitive Assessments for Deteriming Financial Return on Training/Education Experts in the field identify three methods for evaluating training/education offerings that provide a measure of the value of the offering utilizing a cost/benefit approach: 1. Benefit-Cost Benefit-Cost Ratio, 2. Return on Investment, and 3. Forecasting. The information provided by using these models can be used to compare one offering to another based on its contributions to overall return to the organization. The methodology for Benefit-Cost Ratio and Return on Investment is similar in that both require determination of costs and benefits and using that data to calculate the ratio or return. However, the results in the calculations are expressed differently and lend themselves to alternative presentations. presentations. For example, the results from Benefit-Cost Ratio analysis can be presented in a ratio but also are easily communicated in narrative. The results from Return on Investment analysis are always presented in percentages.
2
Introduction With companies scrutinizing scrutinizing costs and tightening purse strings, how do they prove that training/education is worth the investment? A first rate evaluation study may be necessary to justify the investment of time and capital in a training/education training/education offering. The classic model to evaluate training/education offerings offerings developed by Kirkpatrick (1998) looks at four levels of assessment: •
Student reaction,
•
Learning,
•
Behavior, and
•
Business results. When looking at the business results of training, training, many critics say the Kirkpatrick model
does not go far enough in analyzing the impact of training/education on a company’s bottom line. Training/education offerings should be evaluated at a fifth level-level--financial financial return. return. There are several methods for evaluating training/education offerings based on financial return and these methods include both quantitative and qualitative assessments.
Quantitive Assessments for Deteriming Financial Return on Training/Education Experts in the field identify three methods for evaluating training/education offerings that provide a measure of the value of the offering utilizing a cost/benefit approach: 1. Benefit-Cost Benefit-Cost Ratio, 2. Return on Investment, and 3. Forecasting. The information provided by using these models can be used to compare one offering to another based on its contributions to overall return to the organization. The methodology for Benefit-Cost Ratio and Return on Investment is similar in that both require determination of costs and benefits and using that data to calculate the ratio or return. However, the results in the calculations are expressed differently and lend themselves to alternative presentations. presentations. For example, the results from Benefit-Cost Ratio analysis can be presented in a ratio but also are easily communicated in narrative. The results from Return on Investment analysis are always presented in percentages.
2
Benefit-Cost Ratio Benefit-cost Benefit-cost ratio (BCR) analysis allows decision makers to determine the financial return on a training/education program by comparing benefits and costs. BCR is calculated by taking the program benefits and dividing those benefits by the program cost (See Figure 1.).
Figure 1: Benefit-Cost Ratio Formula Program Benefits Benefit-Cost Ratio = --------------------------Program Costs
Program Benefits can be one or more of the following financial gains for training/education: •
Time savings,
•
Increased productivity,
•
Improved quality of output, and/or
•
Enhanced personnel performance.
Program Costs can include the following expenses related to training/education training/education offerings: •
Course development or purchase,
•
Instructional Instructional materials,
•
Equipment and/or facilities,
•
Salaries of instructors and staff, and/or
•
Lost productivity due to training attendance. The result of the calculation is expressed as a ratio. For example, if the BCR calculation
yields a ratio such as 4.5:1, this means for every one dollar of cost invested in training, there will be 4.5 dollars in benefits from the training program. For any BCR that is less than 1:1 such as 0.80:1, the total monetary benefits are less than the total monetary costs and would indicate a less desirable offering. For any BCR that is greater than 1:1, the monetary benefits will be greater than the costs indicating a more desirable training /education offering. The higher the BCR, the greater the return of benefits in relation to the costs. Benefit-Cost Ratio can be used to determine the financial return from a single training/education training/education offering or from two or more offerings (See Figure 2.).
3
Figure 2: Benefit-Cost Ration Comparisons for Three Training/Education Offerings Training/Education Training/Education Offering
Benefit Ben efit-Cost -Cost Ratio
Offerin Offe ring gA
2:1
Offerin Off ering gB
3.5:1
Offering C
8.4:1 8.4:1
The most favorable choice based on benefit-cost ratio analysis would be Offering C which would produce 8.4 dollars in benefits for each dollar of cost. Examples of Benefit-Cost Ratio
1. A financial financial services company purchased software software that would would aid their sales sales force in turning customer contacts into sales. The total cost for the purchase, installation, and training associated with the new software was $83,960. The value of the benefits from increased sales was calculated to be $1,114,940. Program Benefits Program Costs
$1,114,940 $83,960
The benefit-cost benefit-cost ratio in this example is $1,114,940/$83,960 or 13.3:1. In this case, the software seems to be a good investment because each dollar spent in training would produce $13.30 in creased sales or financial benefits. 2. A technology storage storage company was planning to to develop multimedia multimedia training training to improve improve the performance of their technicians. The total cost to develop, install, and train the technicians was $1,748,327. The value of the benefits from increased performance due to training was $1,622,321. Program Benefits Program Costs
$1,748,327 $1,622,321
The BCR is $1,622,321/$1,748,327 $1,622,321/$1,748,32 7 or 0.93 to 1 which is not nearly as high at the BCR in the first example. The training returned only 93 cents in benefits for each $1 in costs; therefore, the company decided that training was not the best way to increase the performance of the technicians.
4
Benefit-Cost Ratio analysis is also useful when there is a need to compare investments in training/education with other investment options available to the organization such as new equipment or technology. A BCR can be calculated for each investment option (See Figure 3.).
Figure 3: Benefit-Cost Ratio for Investment Options Investment Option Training/Education Offering Computer Equipment
Benefit-Cost Ratio 8.5:1 2:1
Based on the comparison, the investment in training/education would result in a higher level of benefits for each dollar invested in comp uter equipment. With the increased competition for each investment dollar, companies are faced with the decision to invest in training or to invest in something else. A well-designed Benefit-Cost Ratio analysis can aid in the decision-making process by allowing several different investment options to be compared to each other. Benefit-Cost Ratio Analysis Worksheet
Gathering and compiling the benefit and cost information needed for the analysis process can be a complicated task. Tables and worksheets are useful tools to facilitating the process. The Benefit-Cost Ratio Analysis Worksheet (See Worksheet 1.) is a sample tool that provides examples of the types of costs and benefits and calculations that can be used in determining a Benefit-Cost Ratio. The worksheet can be completed in three basic steps: identifying benefits and costs and calculating the Benefit-Cost Ratio. Each benefit and cost is specifically defined and assigned a monetary value. The sample worksheet provides examples for benefits and costs commo nly considered by business organizations.
Step One: Benefit Analysis 1. Time Savings The benefits from time saving can come from shorter times for employees to reach proficiency. For example, if a business has new employees in a job, it might take them three weeks to become proficient at a certain job function if they are allowed to learn as they progress through new task or on the job. If training/education was provided to new
5
employees, they may be able to become proficient on the job in one week. The hours saved due to training multiplied by the hourly rate will equal the monetary benefits from time savings. Timesaving benefits can also come from employess learning to perform jobs or tasks more proficiently. The training/education may help eliminate unnecessary steps or help workers make better decisions during the job task. Employees may require less supervision, which frees supervisors to complete other tasks. Also, training may allow employees to manage their time better with hours saved being applied to other productive activities or less overtime. In each case, the time saved multiplied by the hourly rate will produce a monetized benefit. 2. Increased Productivity Additional units may be produced due to a faster work rate. The increased number of units multiplied by per unit dollar value will determine the value of the increased output due to training/education. 3. Improved Quality of Output Improvied quality in the production of products will reduce scrap costs, lost sales, and customer returns. Do llar values can be easily assigned to each one of these items. Improved quality can increase the value of the output by producing increased sales due to higher quality products and can increase a company’s market share. 4. Enhanced Personnel Performance Training/education can reduce accidents and the costs associated with them such as medical claims and lost work. Reducing legal costs associated with workmen’s comp and EEO or OSHA violations can save additional money. If absenteeism is reduced due to training/education then the value of the time saved can be listed. Improved health may be an outcome for employees. The money saved on medical costs and lost time away from the job can also be included. Reduced grievances, claims, and legal actions may be a result of training on sexual harassment or equal employment opportunity, for example. Each of these items can be identified, measured, and monetized.
6
To complete the first step, each benefit is converted to monetary values and then subtotaled. The benefits of training may include something in every category or may only include one item that can be measured.
Step Two: Cost Analysis 1. Course Development or Selection Course development includes the cost incurred for developing training if it is done internally. Costs included here are for doing a needs analysis, researching, designing, creating, writing, and implementing training. Purchase costs are listed if the training/education offering is purchased from an outside vendor. 2. Instructional Materials This item includes all of the costs for any instructional materials that are used by the participants and instructors. This includes any workbooks, handouts, transparencies, videotapes, software, or other materials. 3. Equipment Equipment costs cover all expenses associated with equipment needed for the training/education offering. This can include the cost to purchase or rent projectors, videotape equipment, computers, flip charts, or other training aids. If the equipment is already owned, per use or per hour cost should be listed. 4. Facilities Facilities costs can include rental of classrooms at an off-site location or allocating a “fair share” usage of on-site classroom space that can be allocated as per hour or per day usage rates. 5. Off-Site Expenses Any off-site expenses related to attending or providing the training/education, such as costs for travel, overnight lodging, meals, shipment of training materials, and any other travel costs, are listed. This includes travel by the instructors and/or travel by the participants.
7
6. Salaries The costs for all of the salaries of the participants, instructors, consultants, and support staff are listed. Salaries can be determined from an hourly or daily rate or actual billings from outside providers. 7. Lost Productivity The last item listed under costs is for any lost productivity due to employees participating in training. After all of the costs have been identified and monetized, they are subtotaled. The training/education offering being evaluated may have only one or two costs associated with the development, purchase, or implementation or it could include something in each category.
Step Three: Benefit-Cost Ratio The last step is to calculate the Benefit-Cost Ratio. In the sample worksheet, the BCR is determined by taking the total benefits, $41,000, and dividing them by the total costs, $29,000. The BCR in is 1.41: 1: For every $1 invested in training, there would be $1.41 in benefits.
8
Worksheet 1: Benefit-Cost Ratio Analysis Benefit-Cost Ratio Analysis Worksheet BENEFITS 1. Time Savings Shorter lead time to reach proficiency (hours saved times $) Less time required to perform operations (hours saved times $) Less supervision needed (supervisor hours saved times supervisory $) Better time management (hours freed up times $) 2. Increased Productivity (Quantity) Faster work rate ($ value of additional units, sales) Time saved by not having to wait for help (hours saved times $) Less downtime ($ value of reduced nonproductive time) 3. Improved Quality of Output Fewer rejects (Scrap, lost sales, returns - $ value) Value added to output (Bigger sales, better products - $ value) Reduced accidents ($ value of savings on claims, lost work) Reduced legal costs (EEO, OSHA, workmen's comp. $ value) Improved competitiveness (Change in market share $ value 4. Better Personnel Performance (Attributable to training) Less absenteeism ($ saved) Improved health ($ saved on medical costs and lost time) Reduced grievances, claims, job actions ($ saved) Same output with fewer employees ($ on job s eliminat ed) TOTAL BENEFITS
$ $ $ $
3,000 3,000 3,000 3,000
$ $ $
1,000 1,000 1,000
$ $ $ $ $
2,000 2,000 2,000 2,000 2,000
$ $ $ $
4,000 4,000 4,000 4,000 $41,000
COSTS: 1. Course Development (Time) or Selection (Price, fees) Needs analysis and research Design and creation Writing, validating and revising Producing 2. Instructional Materials Per participant (Expendables, notebooks, handouts, tests) Per instructor (Durables, videotape, film, PC software, overheads) 3. Equipment Projectors, VHS, computers, flipcharts, training aids, other 4. Facilities Rental or allocated "fair share" usage of classrooms 5. Off-site Expenses Travel, hotel overnights, meals, breaks Shipping of materials, rental of audiovisual equipment 6. Salary Participants (Number of hours of instruction time average hourly rate) Instructor, course administrator, program manager. Fees to consultants or outside instructors Support staff (Audio visual, administrative) 7. Lost Productivity Production rate losses or material losses TOTAL COSTS BENEFIT/COST RATIO
$ $ $ $
1,000 1,000 1,000 1,000
$ $
1,000 1,000
$
2,000
$
2,000
$ $
2,000 2,000
$ $ $ $
3,000 3,000 3,000 3,000
$
3,000 $
1.41
9
29,000
Return on Investment (ROI) Return on Investment (ROI) analysis allows decision makers to determine the financial return from training by comparing net program benefits--benefits minus costs--to costs. ROI is calculated by taking the net benefits of training, dividing by training/education costs, and then multiplying the product by 100. ROI is always expressed as a percentage (See Figure 4.).
Figure 4: Return on Investment Formula Net Program Benefits (Benefits - Costs) Return on Investment % = ------------------------------------------------- x 100 Program Costs
Program Benefits can be one or more of the following financial gains for training/education: •
Time savings,
•
Increased productivity,
•
Improved quality of output, and/or
•
Enhanced personnel performance.
Program Costs can include the following expenses for training/education offerings: •
Course development or purchase,
•
Instructional materials,
•
Equipment and/or facilities,
•
Salaries of instructors and staff, and/or
•
Lost productivity due to training attendance. For example, a business used a training/education offering that resulted in $4,000 in
benefits and $1,000 in total costs. Total benefits minus total cost equals a net program benefit of $3,000. Net program benefits ($3,000) are divided by program costs ($1,000) then the product is multiplied by 100 to yield a return on investment of 300% Net Program Benefits (Benefits minus Costs $4,000 - $1,000) $3,000 x 100 Program Costs
$1,000
The Return on Investment calculation can be used when both program benefits and costs can be converted into monetary values. An example of a specific training benefit that can be monetized would be increased productivity. If a trainee can produce 10 additional units per hour
10
as a result of the training and each unit is valued at $20 per unit, the total monetary value of the benefit would be $200 per hour (10 units times $20 per unit). Another example of monetizing benefits could be for improved quality. If, as a result of training, a worker produces 10 fewer defective items per day and if the value of each item was $50, then the monetized value would be $500 (10 units times $50 per unit). For any ROI calcula tions, the higher the percentage, the more desirable the program. For example, if the ROI% is 25, then for every $1 in cost there will be a return of $1 to cover the costs and an additional 25 cents over and above the costs of the program. This is said to have a 25% return on investment. Return on investment should not be used when it is difficult to express the benefits of training in dollar values and percentages. Examples of Return on Investment
1. An electronics systems manufacturing company conducted an 18-week literacy-training program for entry-level electrical and mechanical assemblers. The results of the program were impressive. The benefits of increased productivity and quality yielded an annual value of $321,600. The total costs for the program were $38,233. The calculation indicated a 741% return on the investment in training/education. $321,600 - $38,233 x 100 = 741% $38,233 For each dollar invested, the company realized a 741% return or $7.41 in return after the cost of the program had been recovered. 2. In the second example, the training/education at a large national retail chain yielded a low return on investment. The retail chain attempted to boost sales by conducting an interactive selling skills program for sales associates. The program was developed and delivered by an outside vendor. The total benefits from increased sales were $68,281. The total costs for the program were $32,984. In this example the ROI% calculation is: $68,281 - $32,984 x 100 = 107% $32,984 For every dollar that the company invested in the training, they had an ROI of 107% or $1.07 in return after the costs of the program had been recovered.
11
Return on Investment Analysis Worksheet As with Benefit-Cost Ration analysis, gathering and compiling the benefit and cost information needed for the Return on Investment analysis process can be a complicated task. The Return on Investment Analysis Worksheet (See Worksheet 2.) is a sample tool that provides examples of the types of costs and benefits and calculations that can be used in determining a Return on Investment. The worksheet can be completed in three basic steps: identifying benefits and costs and calculating the Return on Investment. Each benefit and cost is specifically defined and assigned a monetary value. The sample worksheet provides examples for benefits and costs commonly considered by business organizations.
Step One: Benefit Analysis 1. Time Savings The benefits from time saving can come from shorter times for employees to reach proficiency. For example, if a business has new employees in a job, it might take them three weeks to become proficient at a certain job function if they are allowed to learn as they progress through new task or on the job. If training/education was provided to new employees, they may be able to become proficient on the job in one week. The hours saved due to training multiplied by the hourly rate will equal the monetary benefits from time savings. Timesaving benefits can also come from employess learning to perform jobs or tasks more proficiently. The training/education may help eliminate unnecessary steps or help workers make better decisions during the job task. Employees may require less supervision, which frees supervisors to complete other tasks. Also, training may allow employees to manage their time better with hours saved being applied to other productive activities or less overtime. In each case, the time saved multiplied by the hourly rate will produce a monetized benefit. 2. Increased Productivity Additional units may be produced due to a faster work rate. The increased number of units multiplied by per unit dollar value will determine the value of the increased output due to training/education.
12
3. Improved Quality of Output Improvied quality in the production of products will reduce scrap costs, lost sales, and customer returns. Dollar values can be easily assigned to each one of these items. Improved quality can increase the value of the output by producing increased sales due to higher quality products and can increase a company’s market share. 4. Enhanced Personnel Performance Training/education can reduce accidents and the costs associated with them such as medical claims and lost work. Reducing legal costs associated with workmen’s comp and EEO or OSHA violations can save additional money. If absenteeism is reduced due to training/education then the value of the time saved can be listed. Improved health may be an outcome for employees. The money saved on medical costs and lost time away from the job can also be included. Reduced grievances, claims, and legal actions may be a result of training on sexual harassment or equal employment opportunity, for example. Each of these items can be identified, measured, and monetized. To complete the first step, each benefit is converted to monetary values and then subtotaled. The benefits of training may include something in every category or may only include one item that can be measured.
Step Two: Cost Analysis 7. Course Development or Selection Course development includes the cost incurred for developing training if it is done internally. Costs included here are for doing a needs analysis, researching, designing, creating, writing, and implementing training. Purchase costs are listed if the training/education offering is purchased from an outside vendor. 8. Instructional Materials This item includes all of the costs for any instructional materials that are used by the participants and instructors. This includes any workbooks, handouts, transparencies, videotapes, software, or other materials.
13
9. Equipment Equipment costs cover all expenses associated with equipment needed for the training/education offering. This can include the cost to purchase or rent projectors, videotape equipment, computers, flip charts, or other training aids. If the equipment is already owned, per use or per hour cost should be listed. 10. Facilities Facilities costs can include rental of classrooms at an off-site location or allocating a “fair share” usage of on-site classroom space that can be allocated as per hour or per day usage rates. 11. Off-Site Expenses Any off-site expenses related to attending or providing the training/education, such as costs for travel, overnight lodging, meals, shipment of training materials, and any other travel costs, are listed. This includes travel by the instructors and/or travel by the participants. 12. Salaries The costs for all of the salaries of the participants, instructors, consultants, and support staff are listed. Salaries can be determined from an hourly or daily rate or actual billings from outside providers. 13. Lost Productivity The last item listed under costs is for any lost productivity due to employees participating in training. After all of the costs have been identified and monetized, they are subtotaled. The training/education offering being evaluated may have only one or two costs associated with the development, purchase, or implementation or it could include something in each category.
Step Three: Return on Investment The last step to calculate the Return on Investment. In the worksheet, the benefits are listed, monetized, and totaled. The same is done for the costs associated with the training/education offering. The net benefits are determined by subtracting the costs from the benefits. The ROI is then determined by dividing the net benefits, $12,000, by the total costs , $29,000, then multiplying by 100 to produce a percentage of 41.38.
14
Worksheet 2: Return on Investment Analysis Return on Investment Analysis Worksheet BENEFITS 1. Time Savings Shorter lead tome to reach proficiency (hours saved times $) Less time required to perform operations (hours saved times $) Less supervision needed (supervisor hours saved times supervisory $) Better time management (hours freed up times $) 2. Increased Productivity (Quantity) Faster work rate ($ value of additional units, sales) Time saved by not having to wait for help (hours saved times $) Less downtime ($ value of reduced nonproductive time) 3. Improved Quality of Output Fewer rejects (Scrap, lost sales, returns - $ value) Value added to output (Bigger sales, better products - $ value) Reduced accidents ($ value of savings on claims, lost work) Reduced legal costs (EEO, OSHA, workmen's comp. $ value) Improved competitiveness (Change in market share $ value 4. Better Personnel Performance (Attributable to training) Less absenteeism ($ saved) Improved health ($ saved on medical costs and lost time) Reduced grievances, claims, job actions ($ saved) Same output with fewer employees ($ on jobs eliminated) TOTAL BENEFITS
$ $ $ $
3,000 3,000 3,000 3,000
$ $ $
1,000 1,000 1,000
$ $ $ $ $
2,000 2,000 2,000 2,000 2,000
$ $ $ $
4,000 4,000 4,000 4,000 $
41,000
$
29,000
COSTS: 1. Course Development (Time) or Selection (Price, fees) Needs analysis and research Design and creation Writing, validating and revising Producing 2. Instructional Materials Per participant (Expendables, notebooks, handouts, tests) Per instructor (Durables, videotape, film, PC software, overheads) 3. Equipment Projectors, VHS, computers, flipcharts, training aids, other 4. Facilities Rental or allocated "fair share" usage of classrooms 5. Off-site Expenses Travel, hotel overnights, meals, breaks Shipping of materials, rental of audiovisual equipment 6. Salary Participants (Number of hours of instruction time average hourly rate) Instructor, course administrator, program manager. Fees to consultants or outside instructors Support staff (Audio visual, administrative) 7. Lost Productivity Production rate losses or material losses TOTAL COSTS ROI % ($41,000-$29,000)/$29,000
15
$ $ $ $
1,000 1,000 1,000 1,000
$ $
1,000 1,000
$
2,000
$
2,000
$ $
2,000 2,000
$ $ $ $
3,000 3,000 3,000 3,000
$
3,000
41.38%
Benefit Forecasting Investing in training/education in organizations is not an easy decision. For training/education to be successful, purchasers must devote considerable attention to up - front planning for the costs and financial benefits that can be expected from training/education. Benefit forecasting is a before-the-fact method for selecting the best alternative among a variety of training/education offerings. The method is similar to the analytical methods that managers use for deciding which products to develop and which equipment to purchase as well as whether or not to expand or relocate facilities. Investments in training/education can represent major outlays of organizational resources. For this reason, decision makers require information about a training/education program’s expected impact on organizational performance. Lacking sufficient information, decision makers could fail to support just those programs that have the greatest potential for producing significant benefits for the organization. Financial forecasting is potentially the most effective method that can be developed. If the results of the training/education program are forecast in terms that the organization leaders can understand, the training/education program will likely gain the decision maker’s enthusiastic support. Human work performance is the key, and valuing that performance in dollars and cents is the method. Forecasting financial benefits entails a few major departures from other standard quantitative methods of analysis for financial returns. 1. The first is the need to recognize that the value of human beings does not depreciate over time. Their value will continue to increase with added knowledge and skills gained through training/education. 2. A second departure is that the focus of the analysis of benefits expected to result from training/education may be quite broad. Many training/education offerings focus on providing knowledge or skills that will allow individuals or groups to produce more efficiently or effectively on the job. In addition, the impact of newly acquired skills will result in changes in performance of the peers or subordinates of the people who actually participated in the training/education activity.
16
3. A third departure is the period of time on which the analysis of future benefits focuses. The analysis of benefits using the forecasting model is focused on the period of time during which several alternative training/education offerings could be impleme nted and bring about the some change in performance. If several training/education offerings can be expected to yield the same performance results, then the key variable in the financial forecasting model will be the length of time each program takes to achieve the desired performance. 4. The fourth and most significant difference from standard cost/benefit analysis is the difficulty of defining the specific benefit to be derived from a particular investment. In the financial forecasting model, the specific benefit to be derived is the value of future changes in performance after the costs to achieve that change are deducted. The differences suggest that cost/benefit analyses of training/education offerings require a third component, namely the value of the additional performance or the change in performance that is expected. This third component in the benefit-forecasting model is called the performance value. Benefit forecasting is calculated by taking the performance value to result from the training/education offering and subtracting the cost of the training/education offering (See Figure 5.).
Figure 5: Benefit Forcasting Performance Value _________ - Cost Benefit When looking at performances that will change as a result of training/education offering, the analyst focuses on the base- line performance level that is the measure of performance prior to implementing any training/education. The desired or forecasted performance level is the level of performance expected to occur as a direct result of the training/education offering. The gain in performance from the base line to the desired level will be the value used to determine the performance value.
17
It is relatively easy to evaluate the effectiveness of a training/education offering once it has been completed. It is often quite difficult, however, to evaluate and choose the best among several training/education offerings for achieving a desired performance goal. It may be even more difficult to choose between several training/education offerings and other non-training options available to organizations to increase performance. Evaluation examines the effects of a program after it has been implemented. Forecasting develops economic data for making choices between a variety of options before they are implemented. The financial forecasting model supports a forward looking decision- making process. The usefulness of any forecasting model in the decision-making process is dependent on its ability to convey quantities of data to the decision-makers in a way that is easily understood. The financial forecasting model focuses on hard data. Hard data is highly valued by decision makers because they appear to be accurate and precise. Hard data have the following characteristics: •
Easy to measure and is quantitative,
•
Relatively easy to assign dollar values,
•
Based on objective criteria,
•
Often already being used as measures of performance, and
•
More credible in the eyes of management. The importance of using hard data for forecasting purposes when evaluating
training/education offerings is the data that decision makers are accustomed to using when making other investment decisions. Examples of Benefit Forecasting
1. A service organization in a fast moving and highly competitive market wanted to improve customer service in an effort to improve sale volumes and increase market share. The following benefit forecast was developed for a training program to improve customer service. Performance Value
$200,000
-Cost
50,000
Benefit
$150,000
18
The forecasted benefit was significant. A four-to-one return on investment for the training/education offering could be easily realized in 12 months. Based this data, the request for training was approved. An evaluation at the conclusion of the program revealed an actual value of improved performance of $400,000 for a $50,000 investment or a return on investment of 8 to 1. 2. Another example is a manufacturing firm that wanted to train its workers to develop better decision- making skills to be applied during the manufacturing process. The desired outcome was increased production and decreased scrap. The forecasting model developed for this program is as follows: Performance Value (Increased product+decreased scrap) - Cost of decision- making skill training Benefit
$169,700 18,000 $151,700
The forecast increase in performance of $169,700 on an investment of $18,000 will produce a forecasted return on investment of 9.4 to 1. Calculating Performance Values
To calculate the value of improved work performance, four pieces of information need to be gathered: •
Unit of work performance. Some clearly definable unit of work performance is always present in any task or job role.
•
Performance levels. Both the existing level of performance and the target level of performance to be reached in a designated period of time must be determined.
•
Value of each unit. Each defined unit of work performance must be valued in dollars and cents.
•
Performance value. Performance value is obtained by multiplying the value of one unit of work performance times the total number of units that can be attributed to the training/education program. Performance Value Analysis Worksheet
Once the information is gathered, performance values for training/education offerings can be prepared for comparison. Worksheets such as the Performance Value Analysis Worksheet (See Worksheet 3.) aid in the calculation of the performance value for each training/education offering.
19
Worksheet 3: Performance Value Analysis
Performance Value Analysis Worksheet Note: Performance and time units for all offerings must remain consistent throughout the forecast analysis. Option name 1
2
3
Data required for calculations: (a) What unit of work erformance are ou measurin ? (b) What is the performance goal per worker/work group at the end of the trainin /education ro ram? (c) What is the performance per worker/work group at the be innin of the trainin /education ro ram? (d) What dollar value is assigned to each performance unit? (e) What is the development time required to reach the expected performance level? (f) What is the evaluation period?(Enter the longest time (e) of all o tions bein considered.) (g) How many workers/work groups will participate in the trainin /education ro ram?
Calculations to determine net performance value: (h) Will worker/work group produce usable units during the trainin /education ro ram? If no enter 0. (I) What total units per worker/work group will be produced during the development time? (h x e) (j) How many units will be produced per wo rker/work rou durin the evaluation eriod ? [(f-e)xb]+I (k) What will be the value of the worker's/work group's erformance durin the evaluation eriod? ( x d) (l) What is the performance value gain per worker/ work rou ? [k-(c x d x f)] (m) What is the total performance value gain for all for all workers/work rou s? (l x )
Calculating Costs
To calculate the costs used in forecasting benefits, costs are defined as an item of outlay incurred in the operation of a business enterprise, such as for development or purchase of training/education offerings. The following guidelines may help in gathering cost data. •
Costs should be figured in a way consistent with the Performance Value analysis.
•
Record any assumptions and bases for cost consideration.
•
Forecasting is a big-picture operation. Do not be concerned with gathering and recording all the small, insignificant costs.
20
•
Remember that the audience may consist of people from various levels in an organization. Identify which costs they are interested in and the level of information detail they require.
•
Use realistic cost estimates.
•
Do not leave out a cost that is considered critical to the success of the offering.
•
Do not count any costs more that once.
•
If projecting costs far into the future, add an estimation factor.
•
Keep good records of costs. Cost Analysis Worksheet
Cost Analysis worksheets (See Worksheet 4.) are important tools to aid in the gathering and calculation of the information needed for forecasting.
21
Worksheet 4: Cost Analysis
Cost Analysis Worksheet Option name 1
2
3
Analysis: Needs assessment Work analysis Proposal to management Other Other
Design: General Program Design Specific Program Design Other Other
Development: Draft and prototype Pilot test and revise Production and duplication Other Other
Implement: Program management Program purchace or delivery Participant costs Other Other
Evaluation: Program evaluation and report Performance follow-up Other Other
Total training/education program costs
$
22
-
$
-
$
-
After the performance values and costs for each training/education offering have been determined, the data can be combined in a format consistent with the evaluation formats used by the organization for which the analysis is being done. A Benefit Analysis Worksheet may look similar to the sample provided (See Worksheet 5.).
Worksheet 5: Benefit Analysis
Benefit Analysis Worksheet Offering 1. In-house trainin 2. Offsite training 3. Purchased CBT Performance Value $
210,000 $
322,000
$
378,000
Cost $
5,000
$
14,701
$
25,290
Benefit $
205,000
$
307,299
$
352,710
In the sample Benefit Analysis Worksheet, Offering 3 provides the highest dollar value in benefits and would be the best choice from among the three offerings based on the benefit forecasting.
Cost-Effectiveness Cost-effectiveness analysis assumes that the decision maker faces a set of alternatives from which choices must be made. This method of analysis also assumes that the criteria for making the choice must include not only what will be gained (the effects) but also the value of the resources that will be sacrificed to achieve these gains (the costs). Cost-effectiveness analysis measures costs in the same way that costs are measured when utilizing benefit-cost analysis. The resources that are needed for each training/education offering are specified and assessed according to their market values or another technique that closely simulates their market value. Benefits analysis and effectiveness analysis represent different approaches to the measurement of outcomes. Benefit-cost analysis compares the benefits and costs when the outcomes can be assessed in monetary terms; effectiveness analysis uses comparisons of outcomes that are expressed in some other numerical terms such as test scores.
23
An example of an effectiveness outcome would be a training/education offering that was designed to increase reading proficiency as reflected in higher test scores. Another example would be an offering that focused on improving communication skills where the outcomes could be measured by improvements in test scores. Decision-making skills would be another example where the outcomes could be measure by improvements in scores on tests taken before and after the training. In each of these cases the outcomes of the training/education offerings can be evaluated in terms of effectiveness with relation to a goal, but they cannot be readily translated into monetary values. Effectiveness is assessed by standard evaluation techniques for a common set of goals among the training/education offerings that are considered. For example, the effectiveness of training/education offerings may be evaluated by using tests of achievement in the subjects or skills that are being considered. This may be in percentages that the scores increase, changes in grade level measurement, or any other measurement unit that may be appropriate for the analysis. Those training/education offerings with the lowest cost-effectiveness ratios would be considered to be the most promising with respect to the use of an organization’s resources. Cost effectiveness compares program costs expressed in monetary values with program outcomes that are units of substantive goals achieved (See Figure 6.).
Figure 6: Cost Effectiveness Formula Program Benefits Cost Effectiveness = -------------------------Program Costs
Program Benefits one or more of the following outcomes for a training/education offering: •
Percentage increase in test scores,
•
Grade level improvement on achievement tests,
•
Approval ratings for peer or supervisor appraisals, and/or
•
Enhanced personnel performance.
24
Program Costs can include the following expenses for a training/education offering: •
Course development or purchase,
•
Instructional materials,
•
Equipment and/or facilities,
•
Salaries of instructors and staff, and/or
•
Lost productivity due to employee attendance. The cost effectiveness value is expressed numerically. For example, a training/education
offering designed to improve reading comprehension has a cost of $1,000. It produces a 12 percent increase in test scores on a standardized reading comprehension test. The resulting calculation would be 12 divided by 1000 or .012. The result (.012) can be used to compare this training/education offering with others of different costs and also with different levels of outcomes. Cost-effectiveness analysis can be viewed as an extension of benefit- cost analysis. It is based on the same principles and assumptions and uses the same methods as benefit-cost analysis. Cost-effectiveness analysis is a particularly good method to use to evaluate training/education offerings with similar outcomes without having to reduce the outcomes to monetary values. Example Using Cost-Effectiveness Analysis.
The following example is taken from a study to compare the cost effectiveness of three alternative instructional programs in basic reading for adults (See Figure 7.). The cost per student is listed and the total average weighted score for each alternative is listed.
Figure 7: Comparison of Cost Effectiveness for Three Reading Programs Reading Program A B C
Cost per Student $56 $51 $70
Average Weighted Score 50.50 34.00 55.00
Cost Effectiveness 0.90 0.67 0.79
Program A would be the most cost effective in this example. For every dollar invested in Program A it would produce 0.90 of the average weighted score.
25
Cost-Effectiveness Analysis Worksheet
The sample worksheet shows how cost effectiveness can be a valuable tool to help make the best decision when selecting between two alternatives. The example looks at the annual costs and outcomes of two different instructional activities for the improvement of adult basic reading skills and organizes the data using a Cost-Effectiveness Analysis Worksheet (See Worksheet 5.).
Worksheet 5: Cost-Effectiveness Analysis Worksheet Instructional Activities
Program A
Program B
Average number of students per year
130
89
Average instructional hours per year
3,196
4,695
Average number of hours per student
25
53
Average number of hours per grade level change
77
70
Personnel Costs
Licensed teachers with fringe benefits
$
37,413
$
38,459
Educational assistants with fringe benefits
$
7,713
$
7,815
Secretarial/clerical with fringe benefits
$
4,877
$
1,682
Volunteers
$
11,145
Start-up and training amortization
$
4,468
$
306
Instructional and office space
$
1,975
$
1,028
Maintenance, utilities, insurance, and misc.
$
5,034
$
1,521
Instructional technology hardware
$
1,458
$
2,333
Instructional technology software
$
255
$
1,300
Other instructional material
$
978
$
123
General supplies
$
314
$
254
Other miscellaneous supplies and expenses
$
2,501
$
142
$
1,913
$
1,838
Total basic reading resource costs per year
$
80,044
$
56,801
Average cost per student per year
$
616
$
638
Average cost per actual student hour of instr.
$
25
$
12
Average cost per grade level change
$
1,928
$
847
$
-
Facilities Costs
Equipment and Material Costs
Overhead Costs
Overhead charges from other departments Summary of Costs
26
When student participation and performance data are linked with cost estimates, costs and efficiencies between the two can be compared. The cost per student is $616 and $638 respectively; so, there is not much variation here. The cost per instructional hour is $25 for Program A and only $12 for Program B. This would indicate that Program B is less costly on a per unit basis. The cost effectiveness is clearer when the cost per grade level change is considered. Program A takes $1,928 to produce the same grade level change as Program B that only takes $847. Therefore, it could be concluded that Program B is over twice as effective as Program A.
27
Qualitative Assessments for Determining the Financial Return on Training/Education Why are qualitative assessments of the return on training important? Consider the following situation. An organization is evaluating two similar trainingeducation offerings. Both offerings are computer-based training that target advanced users of a popular spreadsheet application. The goal of both offerings is to increase the knowledge and skill of the users to a sufficient level to allow them to pass a standardized certification test on this spreadsheet application. These two very similar offerings have been evaluated using cost effectiveness as a quantitative evaluation method. Both programs seem to be equally cost effective. How do decision makers determine which program will most effectively meet the learning requirements of the organization? Do they make an arbitrary choice? In many training/education situations quantitative assessments do not provide enough information to make the best cho ice. Numbers cannot tell the whole story when trying to assess learners’ opinions, attitudes, insights, and reactions about training programs. Decision makers can get additional information for the decision making process in a number of different ways. •
Interviewing previous training program users.
•
Seeking out testimonials from people who have experienced the training and can attest to the value of that training in their organization.
•
Observing the application of the training both in the classroom and in t he workplace.
•
Assessing important documents related to training. These are examples of qualitative assessment as applied to the evaluation of training. Qualitative assessments can fill in the blanks left by quantitative measures. Those blanks
could be filled with information from comments like “more enjoyable”, “easy to follow”, “logical sequence”, “difficult learning”, “not fun”, and “sometimes hard to follow.” When the blanks are filled in with qualitative data, making the choice between two training offerings might just be easier. Qualitative assessments can provide a much more descriptive picture of the training/education offerings than that provided by the numerical and financial data when only quantitative methods are used.
28
Data Collection Methods
Qualitative assessments involve five basic data collection methods. •
Observations involve going to where the actual training is occurring and observing the actions and reactions of the learners. Observations can be obtained from several training programs to establish a baseline for comparison. The observations should provide the most colorful descriptions possible.
•
Interviewing the learners to get their opinions and reactions may reveal the increased value or preference of one offering over another. Interviews that provide information attesting to the benefits and effectiveness of the training can be most helpful because they will convey relevant, timely descriptions surrounding the actual application of the training in a “real world” setting. Interviewing is an interrogative process developed and guided by questioning of individuals.
•
Testimonials are statements made by individuals who have familiarity with the training and can attest to the value of the offering. Testimonials come from people who have either encountered the training offering or who are experts in the area being addressed by the training.
•
Documents that could be used in a qualitative assessment of offerings include employee evaluations, production reports, attendance records, attitude surveys, minutes from meetings, newspaper or newsletter articles, and other documents that describe the activity that occurs in organizations that will be affected by training offerings. Information gathered from these documents may identify increased values and improved behaviors that may be attributable to the training.
•
Visual images that could be collected and evaluated would be pictures, videotapes, film, slides, and audiovisual computer software. These images can be stored and evaluated in depth at any time that is convenient. A visual image of an event can provide substantially more information about that event than that obtained from a verbal or written account of the event.
29
How Qualitative Assessments Differ from Quantitative Assessments
The goal for assessment of return on investment for training/education offerings is to determine which offerings give an organization the most value. Both quantitative and qualitative assessments can meet this goal. However, this shared goal is where the similarity ends. Quantitative assessments use monetary values, numbers, ratios, and percentages to gather data and measure value. All inputs, outcomes, and values are measured numerically. Qualitative assessments seek to determine the best value using verbal and visual images to richly describe each training/education offering. This data is used to compare and contrast similar offerings to select the one that returns the most value to the organization. Qualitative assessment is defined as an inquiry process for understanding a social or
human problem. The process is based on building a complex, holistic picture formed with words and reporting detailed views of participants. The inquiry process is conducted in a natural setting. Quantitative assessments such as benfit-cost ratio and cost effectiveness have limitations. •
They are unidimensional. They capture only those aspects of performance that can be translated into dollars and cents and ignore characteristics that may be important to the organization’s long term success such as loyalty and company image.
•
They have difficulty measuring performance where feedback between action and effect is not immediate or direct. For example, a program designed to develop organizational trust would be very difficult to evaluate on a strictly quantitative basis. The desired outcome would be to increase the levels of trust in the organization, which is very difficult to measure in financial terms. Example of Qualitative Assessment
The Disney organization is frequently identified as a successful organization that relies on training for sharing and reinforcing its culture. The theme park’s employees learn that they are on stage for a performance while at work and are taught the importance of maintaining their persona. The goals, values, and culture of both the individual employees and the Disney organization as a whole are identified, communicated, and enhanced through a comprehensive training/learning environment. The constantly increasing value of the Disney culture has contributed to its position as a leader in the entertainment industry.
30
At Disney, the changes in the goals, values, and culture, as a result of the training, are difficult to assess in economic terms. Because the effects of those training efforts pervade the organization, financial analysis tools alone cannot be used to separate the economic worth of training. Values are important but difficult, if not impossible, to quantify. The data that represents these values cannot be disregarded just because tools have not been developed to measure or control them. The Disney example makes it easy to see why quantitative assessments such as cost-benefit analysis cannot provide all of the information needed and that the inclusion of qualitative assessments of values can significantly enhance the decision- making process. Implementing Qualitative Assessment
How can qualitative assessment be applied to the evaluation of two training/education offerings that produce similar results when the costs, in time and money, are similar? The solution can be fairly simple. Providers of quality training/education offerings should supply references from current/previous users of their programs. The qualitative assessment process begins by finding the users who most likely characterize the organization. The second step is to ask them what they thought about the training and why they chose to purchase it.
Interviewing is one of the most frequently used data collection methods for qualitative assessments. The interview topics to gather information about the value of a training/education offering could be similar to the following: •
Describe what you liked about the training.
•
Describe what you did not like about the training.
•
If you could make changes to the training, what changes would you make?
•
Describe what you learned and how it can be applied to your job.
•
Describe what learners could do after the training that they could not do before.
•
Describe this training/education offering in relation to other similar offerings that you have experienced.
31
The interview topics for computer-based training might be more specific: •
Describe how well the computer software worked with Windows 98 operating system.
•
Describe how well the learners interacted with the computer-based software supplied with the training program.
•
Describe how well the computer-based training software worked on your company intranet.
•
Describe the hardware limitations that you encountered, if any.
The focus of the interviews is to determine the additional value of one offering over the other for the learners. Qualitative assessment is based on six basic assumptions that provide a description of the assessment process and help differentiate it from the quantitative methods. 1. Qualitative researchers are concerned primarily with process, rather than outcomes or products. This means that they are searching for information to answer questions like,"What makes it work?" and "How does it apply to my organization?" 2. Qualitative researchers are interested in meaning–how people make sense of their lives, experiences, and structure of the world. In the case of assessing training/education, the interest is in the meaning or value to the learners and also the meaning or value to the organization. What does it mean to them? Is it of value? 3. The qualitative researcher is the primary instrument for data collection and analysis. Data are mediated through this human instrument, rather than through inventories, questionnaires, calculators, or computers. The person doing the assessment will include his or her knowledge, experiences, opinions, and biases in the assessment. 4. Qualitative assessment involves fieldwork. The evaluator physically goes to the people, setting, site, or institution to observe or record behavior in a natural setting. Qualitative research is descriptive in that the evaluator is interested in the process, meaning, and understanding gained through words or pictures. For many users of training, the highly descriptive nature of qualitative assessment will give them a clear idea of what is going to occur in the training and how it will effect them.
32
5. The process of qualitative research is inductive in that the researcher builds abstractions, concepts, hypotheses, and theories from details gathered. For users of training, this will mean that they will be able to visualize how the training will work in their organization or what it will be able to offer them in terms of greater skills and knowledge . Process Evaluation
One qualitative approach called process evaluation can be used to evaluate training/education offerings. Process evaluations are aimed at clarifying and understanding the internal dynamics of program operations and focus on the following kinds of questions: •
What are the factors that come together to make this program what it is?
•
What are the strengths and weaknesses of the program?
•
How are learners brought into the program and how do they move through the program once they are participants?
•
What is the nature of learner interactions with the instructor or the technology used? Process evaluations most typically require a detailed description of program operations.
Such descriptions may be based on observations and/or interviews with staff, learners, and program administrators. Many process evaluations focus on how the program is perceived by participants and by staff. Process evaluations permit decision makers and program users to understand the dynamics of a particular training/education offering. Process evaluations are particularly useful in permitting people not intimately involved in the program (for example, those providing funding, public officials, and external agencies) to understand how a program operates. This permits such external persons to make more intelligent decisions about the evaluation, purchase, or use of a single training/education offering or a comparison among several offerings. By understanding the dynamics of program processes and by studying descriptions of the training/education offerings, it is possible to isolate critical elements that have contributed to program successes and failures.
33
When Qualitative Assessments Are Appropriate
The checklist of questions adapted from the work of Patton (1980) can be used to help decide if qualitative assessments are an appropriate evaluation strategy (See Figure 7.). If the answer to any question is “YES”, then the collection of some qualitative data is likely to be appropriate.
Figure 7: Checklist for Determining If Qualitative Methods Are Appropriate CHECKLIST OF ASSESSMENT SITUATIONS WHERE QUALITATIVE METHODS ARE APPROPRIATE o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o
1.
Does the offering emphasize individual outcomes; for example, different participants are expected to be affected in qualitatively different ways? And is t here a need or desire to describe and evaluate these individualized learners?
2.
Are decision- makers interested in clarifying and understanding the internal dynamics of programs –program strengths, program weaknesses, and overall program processes?
3.
Is detailed, in-depth information needed about certain client cases or program sites -particularly successful cases, unusual failures, or critically important cases--for programmatic, financial, or political reasons?
4.
Is there interest in focusing on the diversity among unique qualities exhibited by individual learners?
5.
Is information needed about the details of program implementation to inform decision makers as to what is going on in the pr ogram and how it has developed?
6.
Are program staff and other decision makers interested in the collection of detailed, descriptive information about the program for the purpose of improving the program?
7.
Is there a need for information about the details of program quality, for example, descriptive information about the quality of program activities and outcom es, not just levels, amounts, or quantities of program activity and outcom es?
8.
Will the administration of standardized measuring instruments be overly obtrusive in contrast to the gathering of data through natural observations and o pen-ended interviews? Will the collection of qualitative data generate less reactivity among participants that the collection of quantitative data?
9.
Is there no valid, reliable, and believable standardized instrument available to measure the particular program outcomes for which data are needed?
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
o
Yes
o No
10. Are decision makers interested in having evaluators conduct program site visits to gather information? 11. Are the goals of the program vague, general, and nonspecific? This situation indicates the possible advantage of gathering data about what effects the program is actually having on learners?
34
o
Yes
o
No
12. Is there the possibility that the program may be affecting learners in unanticipated ways?
o
Yes
o
No
13. Has the collection of quantitative data become so routine that no one pays much attention to the results anymore? This suggests a possible need to break the old routi ne and use new methods to generate new insight about the program.
o
Yes
o
No
o
Yes
o
No
o
Yes
o
No
14. Is there a need and desire to personalize the evaluation process by using research methods that require personal, face-to- face contact with the program, methods that may be perceived as “humanistic” and personal because participants are not labeled and numbered, and methods that feel natural, informal, understandable to participants? 15. Do decision makers and information users have philosophical or methodological biases that lead them to prefer qualitative methods, thus increasing the likelihood that they will find the results of a qualitative evaluation particularly believable, credible, understandable and useful? 16. Are decision makers and evaluators interested in increasing their understanding of the program by developing an explanation or theory of program actions and effects that are derived from a holistic picture of the program?
35
Mixed Methods Assessment for Determining Financial Return on Training/Education Mixed methods of assessment for the return on training/education use both a quantitative method and a qualitative method to evaluate the training program. Mixed methods are used in situations where the use of only one method does not supply adequate or compelling information to make the decision regarding the adoption or rejection of the proposed training/education offerings. Advantages for Mixed Methods Assessment
1. The use of qualitative data can broaden the base of the evaluation from a unidimensional financial analysis to include other factors such as loyalty, company image, attitudes toward change, customer service, and long-term organizational goals. Using only one way to see things is simultaneously a way of not seeing other things. 2. Mixed methods helps ensure that any differences between the value of the offerings being evaluated is reflective of the offerings themselves and not of the evaluation methods. 3. The agreement in the results of the two methods in a mixed method approach adds an increased level of “believability” to the data. 4. Using a mixed method approach will also help minimize any weaknesses that may be a part of one or both methods. 5. The accuracy of the assessment can be improved by collecting different kinds of data bearing on the training/education offerings being assessed. 6. Using mixed methods can highlight contradictions and allow fresh perspectives to emerge. 7. Qualitative data can “shed some light on” the meaning of quantitative data. Quantitative data can add a concrete, measurable dimension to qualitative assessments. Examples of Mixed Methods
1. A manufacturing organization was using the quantitative method of Forecasting Benefits to assess the value of investments made in training/education of their employees. The manufacturer needed to train its employees in methods necessary to meet new guidelines and procedures established by the Environmental Protection Agency (EPA).
36
The decision makers were comparing an offsite training program offered at a local college and a computer-based training program purchased from a vendor. The subject matter contained in each program is very similar, only the methods of delivery are different. They determined the performance value, detailed the costs, and developed a Benefit Analysis Worksheet (See Worksheet 6). Worksheet 6: Benefit Analysis
Benefit Analysis Worksheet Offerings 1. Offsite training
2. Purchased CBT
Performance Value
$
316,000
$
328,000
Minus Cost
$
14,701
$
25,290
Benefit
$
302,299
$
302,710
The Benefit Analysis shows that the choice of either alternative will produce approximately the same monetary benefits for the organization. In this case, the decision makers feel they need something more. Qualitative methods can be used to add additional value to the analysis process. In this case the decision makers interviewed employees from other organizations similar to theirs who have used both the offsite training and the computer-based training. Their evaluation of the interviews most strongly supported the purchase of the computer-based training. 2. The same principles can be applied to organizations that may favor qualitative assessment methods to assess training/education. An organization is faced with making a choice between two vendors (Vendors A and B) that can supply computer-based training to develop expertise in a specific technical drawing application used in the engineering department. The decision makers gathered information from other locations in their own organization that had used both of the programs with equal success. They gathered their information from
37
interviews with previous users of both programs and by observing the application of the training in real life work settings. Their analysis of the qualitative data (interviews and observations) showed that the training program supplied by Vendor A had a slight preference by the users and that the application of the training on the job was most easily achieved using the program from Vendor A. However, the qualitative data was not sufficient to determine the impact of the training/education offerings on the organization's "bottom line." So they gathered financial data to calculate the Return on Investment to assist them in the decision making process (See Figure 8.).
Figure 8: Return on Investment Comparison for Two Vendors Vender A $321,600 - $38, 233 ------------------------------ x 100 = 741% Return on Investment $38, 233 Vender B $318,281 - 82,984 ------------------------------ x 100 = 283% Return on Investment $82,984
The addition of the quantitative data from the Return on Investment analysis provides the additional information necessary make a clear cut and rational decision that the computer based training program should be supplied by Vendor A. Worksheet for Mixed Method Analysis
As with other methods for analyzing return on training/education offerings, worksheets are useful tools for organizing and comparing the quantitative and qualitative data for each offering (See Worksheet 7.). The methods chosen for the analysis are dictated by the offerings being evaluated and the organization's requirements for information.
38
Worksheet 7: Mixed Method Analysis Mixed Method Analysis Worksheet Title of training/education offering:
Quantitative Assessment Data
Qualitative Assessment Data
Select the best quantitative method based on the data available, needs of the organization, and the characteristics of the training/education being evaluated.
Select the best qualitative method based on the information available, the needs of the organization, and the characteristics of the training/education being evaluated.
1)
a.
b.
Cost Effectiveness a. What is the performance objective of the offering?___________________ b. What is the numerical measure rating or score of the units of performance improvement?______________________ c. What is the financial cost to achieve the offering objectives?___________________ d. What is the Cost Effectiveness ratio? (b. Divided by c.)_________________________ Select BCR or ROI% a. What is the performance objective of the offering?___________________ b. What is the dollar value of the performance improvement (Benefits)?_______________________ c. What is the dollar value cost to achieve the offering objectives (Costs)?_________ d. What is the Benefit Cost Ratio (b. divided by c.)?_______ e. What is the Return on Investment % ( b. minus c; then divided by c.)_______________ Forecasting Financial Benefits a. What is the performance objective of the offering?___________________ b. What is the dollar value of the performance improvement (Performance Value)?_______________________ c. What is the dollar value cost to achieve the offering objectives (Costs)?______ d. What is the Financial Benefit (Performance Value minus Cost)?___________________
1) a. b.
2) a. b.
3) a. b.
4) a. b.
Observations What values can be determined from observing the offering itself?___________________ What values can be determined from observing the application of the knowledge and skills obtained in the offering in the application or work setting?_________________________________ Interviews Who can be interviewed to attest to the value of the offering?_____________________ What important values or themes can be determined from these interviews?_____________________ Testimonials Who can provide testimonials to the value of the offering?____________________ What values can be determined from the testimonials?__________________________ Records associated with the training/education What records that relate to the value of the offering are available for review?___________________ What values can be determined from using these records to evaluate the offering?_______________
5) a.
Visual images Are there any visual images available to evaluate the offering?_________________ What values can be derived from these visual images to assess the offering?___________
Summary of Mixed Method Assessment
39
Example Using Mixed Method Worksheet
Mixed Method Analysis Worksheet On-the Job Training for Maintenance Technicians
Quantitative Assessment Data Return on Investment a. What is the performance objective of the offering? Improve Maintenance Mechanics Diagnostic and Repair skills for computer related problems b.
What is the dollar value of the performance improvement (Benefits)?_$201,000
c.
What is the dollar value cost to achieve the offering objectives (Costs )?_$199,000_
d.
What is the Return on Investment % ( b. minus c; then divided by c.) _(201,000/199,000)/199,000 X 100 = 1.01%
Qualitative Assessment Data Interviews a. Who can be interviewed to attest to the value of the training/education? Maintenance Team Leader, Maintenance Technician, and Production Supervisor at XYZ Corp. Mr. James Johnson - Maintenance Team leader for first shift at XYZ Corp.: “The OJT currently used at this facility allows for the development of hands on experience through direct interaction with the equipment and the problems associated with them. The process does take a substantial amount of time as most problems encountered are unique to each machine and process.” Mr. Lynn Boman - Maintenance Technician XYZ Corp.: “The OJT at XYZ gives me the opportunity to learn by doing. Once I encounter a new problem and am able to repair it, I can more easily recognize that problem the next time it is encountered. However, the urgency needed to return the machines to production prohibits in-depth investigation and analysis of some problems. The OJT can also be very frustrating and difficult .” Ms. Caroline Rustman -Production Supervisor XYZ Corp.: “The OJT for the maintenance technicians does produce well developed skills. However the length of time that it takes for the maintenance personnel to become proficient in the diagnosis and repair of computer related problems on the new equipment adds substantially to downtime and lost productivity.”
c. What values can be determined from these interviews? Hands on experience, time consuming, frustrating, difficult, learning by doing, prohibits in depth analysis of problems, produces well developed skills, lengthy, substantial downtime, lost productivity.
40
Structured Classroom Training for Maintenance Technicians
Quantitative Assessment Data Return on Investment a. What is the performance objective of the offering? Improve Maintenance Mechanics Diagnostic and Repair skills for computer related problems b.
What is the dollar value of the performance improvement (Benefits)? $417.000_
c.
What is the dollar value cost to achieve the offering objectives (Costs)? $416,300__
d.
What is the Return on Investment % ( b. minus c; then divided by c.) (417,000-416,300)/416,300 X 100 = 0.17%
Qualitative Assessment Data Interviews a. Who can be interviewed to attest to the value of the offering? Engineering and Maintenance Manager, Maintenance Technician, and Production Team Leader, all at XYZ Corp. Ms. Janette McCabe - Engineering and Maintenance Manager at XYZ Corp.: “The addition of continuous training for the maintenance personnel will produce a more highly skilled workforce. In addition, it will be an additional benefit that the company can offer to attract prospective maintenance employees. The training will aid in employee retention through job enrichment and employee development. The classroom training demonstrates the commitment of the company to the long-term investment in their workforce.” Mr. Timothy Ernst - Maintenance Technician at XYZ Corp.: “The classroom training shows that the company is willing to invest in my skills for the longterm benefit of myself and the company. I feel more like a member of the production "team" and not just an expense of operating the plant. The comprehensive, offsite training will allow me to develop a more complete set of diagnostic skills needed to work with the new equipment. Skills gained through OJT are developed on random basis as skills are developed only as problems arise.” Mr. Patrick Hands - Production Team Leader Second Shift - XYZ Corp.: “The offsite training for the maintenance people will really give a boost to the production department . If they can learn to detect and repair problems more quickly, that will lead to less downtime for repairs and higher productivity. The production people will have more confidence in the maintenance people and will be more willing to work with them on a cooperative basis.”
b. What values can be determined from these interviews? More highly skilled, workforce, additional benefit to offer prospective employees, increased employee retention, demonstrates commitment, increased teamwork, more complete skills, less downtime, higher productivity, more confidence, and increased cooperation.
41
Summary of Mixed Methods Assessment The Return on Investment for the on the job training for the Maintenance Technicians is 1.01% The Return on Investment for the structured classroom training for the Maintenance Technicians is 0.17% The interviews with the selected personnel at the plant indicate the OJT did pro duce well developed skills thr ough hands on experience, however, there were some undesirable qualities that were revealed in the interviews that did not show up in the ROI analysis. The OJT was describe as being very lengthy, that it increases downtime leading to lost productivity, and that it prohibits in depth analysis of problems. It was also described as being a most difficult and frustrating method of training. It is the combination of both analysis methods that will allow the best d ecision to be made. If only the ROI analysis is used, the wrong choice will be made. The addition of the interview d ata gives insight into the human side of the training. The people directly involved with the training do not like it or enjoy it. This would indicate that f urther investigation and evaluation is needed. The interviews with the selected personnel at the plant indicate the structured slassroom training produces additional values that were not discovered in the quantitative portion of the assess ment. The addition of the values from the interviews will give subst antial support for the selection of the classroom training as the best choice. This choice would not have been made based on the ROI analysis alone. Worksheet 8: Example Using Mixed Methods of Analysis
The manufacturing company chose to pursue the development of the classroom training in partnership with the local college even though the ROI was better for the on-the-job training The information gathered from the interviews provided them with additional insights into the training that would provide additional value for their particular organization. In this case, the financial analysis alone would have indicated that they should stick with the on-the-job training. However, the additiona l information gained from the qualitative assessment indicated that the classroom training for the maintenance technicians would have a better fit with the culture and structure company.
42
