1 Item Analysis in the Construction of Test Instrument (Dr See Kin Hai) Item Analysis The item analysis is an important phase in the development of an exam program. program. In this phase statistical methods are used to identify any test items that are not working well. If an item is too easy, too difficult, failing to show a difference between skilled and unskilled examinees, or even scored incorrectly, an item analysis will reveal it. The two most common statistics reported in an item analysis are the item difficulty, which is a measure of the proportion of examinees who responded to an item correctly, and the item discrimination , which is a measure of how well the item discriminates between examinees who are knowledgeable in the content area and those who are not.
After you create your objective assessment items and give your test, how can you be sure that the items are appropriate -- not too difficult and not too easy? How will you know if the test effectively differentiates between students who do well on the overall test and those who do not? An item analysis is a valuable, yet relatively easy, procedure that teachers can use to answer both of these questions. To determine the difficulty level of test items, a measure called the Difficulty Index is used. This measure asks teachers to calculate the proportion of students who answered the test item accurately. By looking at each alternative (for multiple choice), we can also find out if there are answer choices that should be replaced. 1. Difficul Difficulty ty Index Index:: to show show the test test items items if they they are: are: i) too difficult, ii) average, or iii) too easy I =
Total No of students who answer correctly (C) Total no of students doing the test (T) 2. If the sample sample size size is large large and to cut down the the time in computatio computationn of the Index, I Difficulty Index = I =
C H
+ C L
T H
+L
Where C = Total no. of students in the High Achievement group that answer correctly H
C L T H
= no. of students in the Low Achievement group that answer correctly
+L
= Total no. of students in the High and Low achievement groups responding
Difficulty Index
Evaluation of item
Conclusion
I < 0.3
too difficult
modify the item
average
accept the item
too easy
modify the item
0.3
≤
I
I > 0.8
≤
0.8
2 The Item Discrimination Index (D)
, refers to how well an assessment differentiates between high and low scorers. In other words, you should be able to expect that the high-performing students would select the correct answer for each question question more often than the low-performing low-performing students. If this is true, then the assessment is said to have a positive discrimination index (between 0 and 1) -- indicating that students who received a high total score chose the correct answer for a specific item more often than the students who had a lower overall score. If, however, you find that more of the low-performing students got a specific item correct, then the item has a negative discrimination index (between -1 and 0). Discrimination Index
Another consideration consideration for an item analysis is the cognitive cognitive level that is being assessed. For example, example, you might categorize the questions based on Bloom's taxonomy (perhaps grouping questions that address Level I and those that address Level II). In this manner, you would be able to determine if the difficulty index and discrimination index of those groups of questions are appropriate. For example, you might note that the majority of the questions that demand higher levels of thinking skills are too difficult or do not discriminate well. You could then concentrate concentrate on improving improving those questions and focus your instructional strategies strategies on higher-level skills. 1. “D” reports reports the differenc differencee between between the proportio proportionn of High High and Low aacxhieve aacxhievers rs in answering answering aa test item correctly 2. D = high shows shows the the item item is good, D = low shows the item item is badly construct constructed ed 3. The purpose of D is used used to distinguish distinguish between between the the high achievers achievers from the the low achievers achievers D = Discrimination Index =
C H
− C L
1 T 2
H + L
Discrimination Index (D) D > 0.4 0.2 0
≤
≤
D
high positive value ≤
0.4
D < 0.2
D<0
Item Evaluation
Conclusion acceptable
average positive value
modify the item
low positive value
reject item and rewrite
negative value
reject
(Low achievers perform better then high achievers)
3 COURSEWORK 1 (Test Construction)
You have constructed a 10 items (questions) test and 20 students have responded. The results are shown in the Table below. 1. Fill Fill in all all the the bla blank nks. s. 2. Conclude Conclude ifif you accept accept the the test item item ( √ ) or reject reject the the test test item item ( X ) at the Conclusion row.
4 COURSEWORK 2
Item Analysis Worksheet Ten students have taken an objective assessment. The quiz contained 10 questions. In the table below, the students’ scores have been listed from high to low (Joe, Dave, Sujie, Darrell, and Eliza are in the upper half). There are five students in the upper half and five students in the lower half. The number“1” indicates a correct answer on the question; a “0” indicates an incorrect answer. Student Name Joe Dave Sujie Darrell Eliza Zoe Grace Hannah Ricky Anita
Total Score (%)
Questions 1
2
3
4
5
6
7
8
9
10
100 90 80 70 70 60 60 50 40 30
1 1 1 0 1 1 0 0 1 0
1 1 1 1 1 1 1 1 1 1
1 1 0 1 1 1 1 1 1 0
1 1 1 1 0 0 0 1 0 0
1 1 1 1 1 1 1 0 1 0
1 1 1 1 1 1 1 0 0 1
1 1 1 0 1 0 0 1 0 0
1 1 1 1 0 1 1 0 0 0
1 0 0 0 0 0 0 1 0 1
1 1 0 1 1 0 1 0 1 0
Calculate the Difficulty Index (p) and the Discrimination Index (D) for each question. Item Question 1 Question 2 Question 3 Question 4 Question 5 Question 6 Question 7 Question 8 Question 9 Question 10
# Correct (Upper group)
# Correct (Lower group)
Difficulty (p)
Discrimination (D)
Answer the following questions: 1.Which question was the easiest? 2. Which question was the most difficult? 3. Which item has the poorest discrimination? 4. Which questions would you eliminate first (if any) – why?
5
