the poli policy cy framework
6
The State Education Standard
The
Purposes &
Elements
of
Effective Assessment Systems by Carla Claycomb and David Kysilko
F
or many yea years, rs, gov governors ernors,, le legis gisla latures tures,, and state boards of education education have struggle struggled d to find the most effective role for the state in improving student achievement and providing prov iding oversight oversight for th e education system. system. Today Today,, nearly every every state is engaged engaged in standards-base standardsbased d reform, as see seen n by the development development of stat statewide ewide academic academic goals and guidelines in core subjects and the subsequent new state tests that will measure students’ student s’ progress toward achievi achieving ng thes th esee goals goals.. T his has prove proved d to be a complex complex,, lengthy lengthy,, and at times tim es frustratin frustrating g and cont controv roversia ersiall undertaking. In focusing on the th e testing aspects aspects of this th is eff effort ort,, NA NASBE’ SBE’ss 1997 Stud Study y G roup on Statewide St atewide Asse Assess ssment ment Sys Systems tems began with with the t he most fundament al questions: D oes standardsstandards-base based d reform make se sense? nse? Is this the best strategy states can use to ensure that all students are well-prepared for the future? 1 The Study Group clearly affirmed the importance of state standards and noted that a host of issues issues had combined to t o make stand standards, ards, in the th e words words of a previous previous NA NASBE SBE report, repor t, “one of 2 the most powerful options for school reform.” M oreo oreov ver er,, despite some some recent recent rough patches in th their eir implementation, implement ation, including various grassroots grassroots backlas backlashes hes and Sec Secretary retary Riley Riley’’s cal calll for
Spring 2000 7
a “midcourse correction,”the vast majority of state board members remain remain committed to standards. standards. T he Study Group G roup laid laid out its reasons for supporting this reform strategy: Standards focus on outcomes, which in turn t urn prov p rovide ide flexi flexibili bility ty to local schools and districts in how to achieve those outcomes; • Standards provide for uniform goals and a uniform measurement tool in an age of increased mobility and wide variations in local grading practices; • Standards are developed through an open process and are publicly available for any teache teacherr, parent, parent, or student to see; • Standards are set at high levels so that students are expected to learn challenging material, material, analyze analyze information, an d communicate results; results; and • Standards apply to all all students, students, not just just the college bound. Th is articl article, e, written written by NASBE staff members who facilitated the work of the committee, committee, is based on the Study Group’s recommendations. It reviews reviews the Study G roup’s conception of t he purposes pu rposes of state assessments assessments and uses these th ese to build a framework for th e essent essential ial characteristics of effeceffective assessment assessment systems. systems. It also includes includes updated information informat ion to reflect recent events in the world of assessment policy.
mented ment ed are indeed indeed fulfilling fulfilling those purposes. Some of the underlying purposes purp oses are as as follows follows:: • Assessments help ensure that standards are taken seriously. W hen t he content cont ent of state stat e assess assessments ments is closely closely aligned aligned with that th at of the standards, this th is is is a clear clear signal to teachers, adminis administrato trators rs,, students students,, and the public public that that th e state state is serious about its academic demic standards standards.. BasiBasically cally, as all all teachers and students student s know, know, if “it’ “it’ss on the tes test,” t,” it mus mustt be import an an t. t. Align in in g assessments with standards is critical for another reason, reason, as well:if well:if material is covered on the assessment but is not in the standards tandards,, it is is inherently unfair to students and teachers alike. In such cas cases es,, the material may not have been covered in class because, agai again, n, no one one knew knew it was important. Not only must content be both in the standards and on the assessment, but the asse assess ss-ment must mean something thing:: there there mus must be be “stakes” stakes” attached to the assessments at some point. point. For students students that point is most likely to be in terms of a diploma; that is, a student ’s level level of achiev achievement ement on state assess assessments ments could be indicated on the th e diploma, or meeting the th e state’s state’s academic stand standards ards could be one requirement for graduation. graduation. For districts districts and schools, schools, it means student scores scores on state assessments assessments should be one component componen t of the t he state’s state’s account accountability ability plan. • Standards and assessments guide teaching and learning. C learly articulated content standards and t he examples examples used in performance standards or curriculum frameworks to illustrate the knowledge and skills embodied in the standards are intend ed to guide teachers toward the th e state’s visio vision n of esse essential ntial teaching teaching and learning. learning. O f course, course, teachteachers will vary in how closely they adhere to the examples in creating creating their lessons lessons.. T he point is, is, when standards are are taken seriously, seriously, all teachers will will share a vision—shaped vision—shaped by a consensus of the best thinking in the state—of the knowledge and skills students should acquire. • Assessments help individual students meet the standards. The ultimate goal of state standards and assessments is not to sort students into the t he “haves”and “haves”and “have “have nots” not s”of of knowledge. knowledge. T he goal is to enable enable all all students students to meet th e standards. standards. In this th is sense, results on state assessments assessments should be part of a support system that can signal when a student is in trouble
The ultimate goal of state standards a nd assessments i s not t o sort students into “haves” a nd “have nots” of knowledge. The goal i s t o enable a l l students t o meet t he standards.
The Purposes of Assessments in Relation to Standards If standards provide the skeleton or framework on which states can build their th eir academic system, system, assess assessments ments are th e muscles that bring the th e system system to life. By now all the states and other jurisdictions that are developing or have developed standards are also aligning their state assessments with the new standards. T his is critical, critical, for the nature natu re of these th ese new assess assessment ment systems may well determine whether the standards truly drive teaching, learning, learning, equity equity,, and accountabili accountability ty in the t he state; whether they are merely vaguely remembered pronouncements that gather dust in districts’central districts’ central offices offices;; or whether th ey are denounced by parents, teachers, teachers, and students alike alike as central central cogs cogs in a factoryfactorymodel education education system system the narrows narrows the curricul curriculum, um, inhibits creativity creativity,, and drives low-achiev low-achieving ing students student s out of schools. schools. W hatever the final assess assessment ment system system in a state looks like, like, policymakers initially need to step back and understand what their purposes for the assessments are and then regularly revisit this discussion discussion over the years years to ensure en sure that the th e assessments assessments as imple-
8
The State Education Standard
•
academically academically and needs extra help or special special instruction. instruct ion. At the th e same same time, teachers should be able able to use the th e standards in developing their own classroom assessments to help identify students who may need more instruction or to inform a teacher when it is time to adjust his or her curriculum or instructional instruction al strat strategies egies.. If states are serious about having all students meet the t he standards, they will will provide provide assis assistance tance to districts and schools for this remediation. Assessments help policymakers policymakers ensure that all students have access to a sound education. Policymakers want to assure the public that all students, students, no matter matt er where they liv livee in the th e state, have have a reasonable reasonable opportunity to get a sound education. education. As indicated above, having assess assessment mentss as one part of th e state’s accountability system helps ensure that the standards are taken seriously. Just Just as important impor tant is using the accountability accountabilit y sys sys-tem to t o verify that th at all schools and districts d istricts are providing a “rea“reasonable opportunity opportun ity.” .” T his is a basic basic state state responsibility— indeed, it is often specifical specifically ly addressed in state constitutions. constitution s. Some have expressed concern that standards and high-stakes assessments will only mean that more of those students who were having trouble in the past will not graduate in the future. Instead, the Study Group believ believes es that standards and and assess assessment ments, s, when part of an overall overall evaluation evaluation and accountability ability system, system, can help provide provide the state with th e information and the mechanisms it needs to bring a solid education within the reach of every student.
Elements of Effective Assessment Systems O ver the years years states have h ave developed developed a wide range r ange of assessassessment systems with with differe d ifferent nt purposes p urposes and consequences, consequences, reflecting reflecting the th e unique content cont ent and character of o f each state’s education system and traditions. Such diversi diversity ty is a unique and vital aspec aspectt of the t he nation’s nation’s heritage of state and local control of education. H owever, owever, the Study Group concluded that there are general characteristics that th at highly h ighly effectiv effectivee state assessment assessment systems share. A brief look at these th ese essential essential element s follows. follows. (1) An Effective State Assessment System is Aligned with Rigorous State St ate Standar St andards. Standards Stand ards need at least least two t wo characteristics characteristics to be effectiv effective. e. First, First , they must be rigorous and detailed enough to guide curriculum and asse assess ssment ment developme development. nt. Second, Second, standards must be put in force via via a closely closely aligned aligned assessment assessment system. system. M any states have developed standards that are not rigorous or specific enough to meaningfully meaningfully direct direct assessment assessment development. development. In these states, states, it is likely likely to be the t he content and structure of their asses assessments, sments, rather than the t he content of their standards, that driv d rivee school school reform. reform. (2) An Effective State Assessment System Addresses Specific Goals and Purposes. W hereas standards standards define what to test, t est, assess assessment ment goals define define who to test, test, when when to tes t est, t, and what type of test test to use. As indica indicated ted above, above, the Study St udy Group felt strongly that a primary purpose for any state assessment system should be to improve teaching and learning. H owever owever,, tests to improve teaching teaching and learning learning can take t ake many different forms and address a variety of complementary goals and purposes. poses. For example example,, tests to inform student student remediation remediation need to be given given to every student in the testing t esting grade, constructed to give
rich information to t o teachers, given given sufficiently sufficiently early enough in the th e year year to inform instruction, and designed for effic efficient ient scoring so that results can be returned to the classroom as soon as possible. Tests to evaluate evaluate schools or districts can be given given to t o a sample of students and n eed not be given every every year year,, which saves saves money and instructional time. Aligning state assessments with state goals and purposes makes good financial financial sense. sense. Assessments Assessments that th at are implemented without clear goals may measure variables that are unnecessary or may measure necessary necessary variables variables in inefficient inefficient ways. ways. For example, example, measuring every every student ’s writing ability to inform district curricular curricular improvement in writing writ ing is “ove “overkill” rkill”;; only a sample of students students need to be tested for that p urpose. urpose. But if a state goal is to ensure that all graduates can write a well-formed paragraph, every every student student must be tested. (3) An A n Effective State S tate Assessment System Balances Validity Validity, Reliability, Reliability, and Efficiency. Efficiency. For assess assessment ment results to mean anything, the test t est must be both valid (meaning that it measures what it purports to measure) and reliable reliable (meaning, (meaning, among other other things, that students would would get similar results results if tested again). Statisticall Stat istically y, it is impossible for for a test to be completely valid valid and reliable, but even making them t hem “suffi “suffi-ciently”va ciently” valid lid and reliable to satisfy stakeholders while at the th e same time being fair and equitable is a challenge. Fundament Fund amentally ally,, those th ose who who write assessments assessments have two types of questions to choose from: from: multiplemultiple- choice choice and performa p erformance. nce. In general, multiple-choice multiple- choice tests are very very reliable reliable provided they include a suffic sufficiently iently large large number of items. M ultiple-choice ultiple-choice tests are also also effic efficient ient in terms of testing time and cost. cost. But they t hey do not resemble resemble good good instruction. instruction. Furthermore, multiple-choice multiple-choice tests essentially measure measure the th e ability to choose th e correct answer, answer, which is not a skill often required by adults in the “real world.” For these reasons, reasons, multiple-choice multiple- choice tests have been criticized for for having low validity. Performance assessments are more likely to measure skills that policymakers policymakers want want students stud ents to develop, develop, such such as the th e ability ability to write a persuasive paragraph or to draw inferences from a science experiment. Constructed Constructed response response ques questions tions,, which which require require students students to write short-essay responses or solve problems and show their work, are currently the most popular performance items used on state tests, though some states have have also also worked with more extended performance tasks tasks and portfolios. portfolios. H owever owever,, experience experience in Kentucky, Kentucky, Maryland, Maryland, and Vermont, Vermont, among other states, states, indicates indicates that the various types of performance assessment differ in their validity and reliability ability.. In general, general, constructed response response items administer administered ed in a trat raditional testing situation are more likely than other types of performance assessment to be scored accurately enough to produce reliable test results. results. O f course course,, they are less less effic efficient ient than th an multiplemultiplechoice questions questions because because of the greater testing time per item, and because more time and money are required to score responses accurately and consistently. Because Because no one test can “do “do it all,” the th e Study Group G roup recommended that th at states consider ensuring fairness fairness and curricular covcoverage and depth by implementing an assessment system with more than one test that balances performance assessment items with multiple-choice multiple-choice.. T he N ational Asses Assessment sment of E ducational ducational Progress (NAE P), the nation’ nat ion’s primary measure measure of what students student s
Spring 2000 9
The Study Group recommended that states The consider...implementing a n assessment system with more than one test that balances performance assessment items with multiple multiple-choice. -choice. know and can do on a state and national n ational level, level, is a good example example of a test that combines multiple-choice and performance items and has proven both valid and reliable for use at the state and national, national, rather than individual, individual, level. level. (4) An Effective State Assessment System Informs Instruction and Has Consequences. Two primary reasons for developing a state assessment system are (1) to inform student instruction by pointing out student learning gaps and students studen ts needing acceleration, acceleration, and (2) to assign assign rewards rewards and sanctions sanctions to students, students, teachers, teachers, schools, schools, or districts districts based upon assessment outcomes. Informing Instruction. The Study Group asserts that once states establis establish h standards, they have both a moral and legal legal obligaobligation to provide all students with the instruction they need to attain them. Assess Assessments ments are instrumental instrument al to diagnose deficienci deficiencies es and to help teachers target instruction for individual individual students. Especially Especially in elementary elementary grades, a state test can be a powerful powerful tool to focus focus attention and resources resources on helping students, schools, schools, and districts meet standards. Unfortunately Unfortun ately, not all states are including including remediation in their assessment assessment agendas: while 40 states require or have plans plans to require remediation remediation for students not meeting standards, standards, only 29 back the requirement with with funds, and only 13 13 require and fund 3 remediation in the four core subject areas. H elping all stud students ents reach their th eir capacity capacity also also requires that teachers receive high-quality information about students who score well on state stat e assess assessment ments. s. O nce states establish establish a system system to encourage encourage all all students to achieve achieve high high standards, standards, they have have an obligation to provide high-achieving students with the instruction they th ey need to develop to their th eir full capacity. capacity. If a student ’s performance on a state assessment points out subject areas in which performance formance exceeded exceeded state expectations expectations,, there should be mecham echanisms in place, place, based upon assessment assessment results and state standards, to encourage that student to continue achieving. Having consequences. Although providing remediation and acceleration to particular students is necessary for an effective accountability system, system, it is not sufficient. To have the th e greatest greatest impact on teac t eaching hing and learning, learning, an assess assessment ment system system also also needs to have consequences consequences.. M aking asses assessments sments count as one component of an accountability plan is important because studies show that high school students perform better on assessments that have consequences for them. Teachers are also more likely to take test results
10
The State Education Standard
seriously and more inclined to change their instructional practices to reflect the content and format of a test if the test has consequences. 4 C urrently, urrently, 28 states tie or plan to tie the award of high school school 5 diplomas to achievement of their standards. In these, these, asse assess ssments ments are commonly commonly used used alone, rather than th an as one component of a multifaceted tifaceted evaluation evaluation system. system. D enying enying diplomas to students based on test scores when students are otherwise qualified to graduate means that students who do well in school but perform poorly on the state assessment may be unfairly penalized by a one-shot evaluation of their accumu accumula lated ted school school work. work. Consequently Consequently,, the Study Group G roup asserts asserts that either: (1) high-stakes high- stakes tests should be only one of seve severral components that are considered when making decisions about student promotions p romotions and graduation; or (2) state assess assessments ments should be used as high-stakes measures in ways other than graduation or promotion, such as for for conferring diplomas diplomas with endorsements. More common than using state assessments to assign consequences to individual students is using assessment results for school school or distric districtt accountabil accountability ity.. H owev owever, er, the same same caution caution applies applies to schools schools and to districts districts as to individual individual students: no school should be penalized on the basis of test results alone. Rather, Rat her, test results should be one component of an evaluation evaluation syssystem that t hat may m ay also include include variables variables such such as dropout rates, rat es, teacher attrition rates, expenditure expenditure information, information, and attendance rates. rates. The Study Group strongly recommended that states guarantee that students have the remediation or acceleration necessary to learn learn to the th e highest highest standards. O ne state board board that is curcurrently taking this issue issue seriously seriously is Maryland, where the board continues to fight fight for its “K-12 “K-12 In tervention” tervention”plan, plan, a proposal proposal to create a $49 million remedial education program to help students keep up with the learning expectations set out in the state standards and assessments. assessments. T he board has signaled signaled to the th e govergovernor and legislature that if there is not sufficient funding for this program, they th ey will have to consider consider alternatives that th at basically basically water down the t he standards stand ards by lowering lowering the passing passing grade or eliminating passing state tests as a requirement for graduation. 6 (5) An Effective State Assessment System Has Mechanisms to Encourage Schools and Districts Dist ricts to Align Their T heir Instruction Instruction and Evaluation with the State System. Standards and assessments should serve as a catalyst to direct local local reforms reforms in improving improving teaching and learning. M ost
states seem to assume that instituting standards and assessments will prompt school districts to align their practices with them. th em. Research Research suggests suggests that such an assumption may be warranted for forward-thinking districts that have the capacity to tackle change head-on . Schools that th at lack lack the will or capacity capacity to change, however however,, are unlike unlikely ly to benefit benefit from state mandates alone. Explicit programs linking state standards and assessments with district and classroom teaching and assessment practices are critical to promote student achievement in all schools and districts. O ne effective effective way way to impact class classroom room teaching and assessment practices is to include performance items on the state asses assessment. sment. In general, teachers are are more likely likely to incorporate the goals and skills of a performance assessment in their instruction, but t hey need help to do so effec effective tively ly..
Conclusion W hen state policymakers policymakers begin building an effective effective state state assess assessment ment system, they th ey are embarking on a complex and timet imeconsuming consuming proces process. s. T his is because because,, perhaps even even more than most other oth er policy issues, issues, the process of building a standards-led education system system does not end for policymakers policymakers when the th e ink on the th e policy policy dries. NASBE’ NA SBE’ss Study Group was emph emphatic atic that state boards need to monitor the implementation of standards and assessment policies to ensure that their policy vision is realized. T his means, means, minimally minimally,, that substantial substantial convers conversations ations about standards and assessments should appear with regular frequency on the board agenda. agenda. Preferably Preferably, these conv convers ersations ations should should compare progress in assessment development and implementation with the criteria listed above to ensure that the assessment is as effective as possible. Best of all, policymakers should be clear in defining th e type of asse assessments ssments they envision. envision. State policymakers that proactively and specifically delineate key characteristics of their state assessments are in the best position to ensure that th at they t hey drive a sys system tem that t hat fosters high achievement achievement for all students entrusted to their care.
(6) An Effective State Assessment System Has a Clearly Articulated Articulated Relationship with N ational Measures Measures of Student Performance. As states invest time and resources in developing a comprehensive hensive assess assessment ment system, system, they must ensure they can calibrate Carla Clayco Claycomb is an educ educati ation on rese researcher and writ w riter er.. D avid av id assessment results with the performance of students across the editor or of the State Education Standard. In I n 1997 19 97 the t heyy staffed staffed country. country. C urrently, urrently, most states states that have articulated articulated standards Kysilko is edit AS BE’’s Study St udy Group on on Statew St atewide ide Asses Assessment sment Systems. have designed assessment programs unique to their state. N ASBE Because Because of of this, states cannot cannot compare their performance or that of individual individual students, students, schools schools,, or districts districts with those of 1. Study Group on Statewide Assessment Systems, T he Full Mea M easure sure(Alexanother states and have no quantifiable way of demonstrating dria, VA: The National Association of State Boards of Education, 1997). Standards-B ased Educ E ducaathat th eir students students perform perform to national standards standards.. T he Study 2. J. Kendall and R. Marzano, T he Fall and R ise of Standards-Base tion (Alexandria, VA: The National Association of State Boards of EducaGroup contended that being serious about standards and tion, 1996). assess assessments ments means ensuring th at standards define, and assessassess- 3. H. Glidden, Making Standards Matter 1999 (American Federation of Teachers, 1999). ments measure, world class class performance. 4. R. Truby, unpublished presentation to the NASBE Study Group on The best national benchmarks available for standards and Statewide Assessment Systems, 15 March 1997. Standards M atter 1999. assessments are the NAEP tests and NAEP subject frame- 5. H. Glidden, M aking Standards Standards,”” Washington Post , 8 March March 2000. 2000. works. works. T hese frameworks frameworks and assessme assessments nts reflect reflect a general 6. “Fund the Standards, consensus about important competencies and expected levels of performance at key ages and in core subjects. subjects. Although N orth C arolina arolina is a notable exception, exception, most state standards and assess assessments ments are not aligned with the NAEP subject frameworks, tests, tests, or standards. standards. As a resul result, t, states states that participate participate in NAEP may have widely differing results on national national and state indicators indicators.. For example example,, whereas whereas 35 percent percent of Wisconsin fourth- grade students were “profic proficient” ient” on t he 1996 NA EP math assess assessment, ment, 88 percent percent of fourth graders were proficie proficient nt on W isconsin’ consin’s own state exam. exam. In L ouisiana, ouisiana, seven seven percent were were “prof “profic icient” ient” on N AEP, whereas whereas 80 80 percent percent of fourth graders were were “proficient” proficient” according according to the state assess assessment. ment. M any states states reporting results from from different programs run into the problem of seemingly inconsistent results because the two programs have different frameworks and performance standards. Reporting Reportin g such such widely divergent divergent results results can be confusing to parents and the general public and send mixed messages messages to teachers t eachers and administrators concerning what content and skills to teach.
Even more than most other policy issues, t he process of building a standards-led education system does not end for policymakers when the ink on t he policy dries. Spring 2000 11
