2009_DOF_Manual[1]

ASEB SEBA A Manu nua al fo forr th the e A Dir ire ec t Obs Obse er va vati tion on For orm m

St eph pha ani nie e H. Mc Conaug Conaughy hy & Thomas Tho mas M. Ac Achenbach henbach

l t h e f o A r S E t B h A e D A i S r E e B c A t D O i b r s e e c r t v a O t b i o s n e r v F a o t r i m o n F o r m

brary of Congress Control Number: xxxxxxx ISBN

978-1-932975-12-3

M c C o n a u g h M y c C & o n A a c u h g h e n y b a & c h A c h e n b a c h

l t h e f o A r S E t B h A e D A i S r E e B c A t D O i b r s e e c r t v a O t b i o s n e r v F a o t r i m o n F o r m

brary of Congress Control Number: xxxxxxx ISBN

978-1-932975-12-3

M c C o n a u g h M y c C & o n A a c u h g h e n y b a & c h A c h e n b a c h

Manual for the ASEBA

Direct Observation Observation Form

Stephanie H. McConaughy, University of Vermont & Thomas M. Achenbach, University of Vermont Vermont

Ordering Information

This Manual and other ASEBA materials can be ordered from: ASEBA 1 South Prospect Street Burlington, VT 05401-3456

Fax: 802-656-5131 E-mail: [email protected] Web: www.ASEBA.org

Proper bibliographic citation for this Manual :

McConaughy, S. H., & Achenbach, T. M. (2009). Manual for the ASEBA Direct Observation Form. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Related Books

Achenbach, T.M., (2009). The Achenbach System of Empirically Based Assessment (ASEBA): Development, Findings, Theory, and Applications. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Achenbach, T.M., & McConaughy, S.H. (2009). School-Based Practitioners’ Guide for the Achenbach System of Empirically Based Assessment (ASEBA) (6th ed.). Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Achenbach, T.M., Pecora, P.J., & Wetherbee, K.M. (2009). Child and Family Service Workers’ Guide for the Achenbach System of Empirically Based Assessment (ASEBA) (6 th ed.). Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Achenbach, T.M., & Rescorla, L.A. (2001). Manual for the ASEBA School-Age Forms & Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Achenbach, T.M., & Rescorla, L.A. (2009). Mental Health Practitioners’ Guide for the Achenbach System of Empirically Based Assessment (ASEBA) (6 th ed.). Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Achenbach, T.M., & Rescorla, L.A. (2007). Multicultural Guide for the ASEBA School-Age Forms & Pro files. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families. Achenbach, T.M., & Rescorla, L.A. (2007). Multicultural Understanding of Child and Adolescent Psychopathology: Implications for Mental Health Assessment. New York: Guilford Press. Achenbach, T.M., & Ruffle, T.M. (2007). Medical Practitioners’ Guide for the Achenbach System of Empirically Based Assessment (ASEBA) (5th ed.). Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. McConaughy, S.H. (2005). Clinical Interviews for Children and Adolescents: Assessment to Intervention. New York: Guilford Press. McConaughy, S.H., & Achenbach, T.M. (2001). Manual for the Semistructured Clinical Interview for Children and Adolescents (2nd ed.). Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. McConaughy, S.H., & Achenbach, T.M. (2004). Manual for the Test Observation Form for Ages 2-18. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Copyright 2009 S.H. McConaughy & T.M. Achenbach. All rights reserved. Unauthorized reproduction prohibited by law. ISBN 978-1-932975-12-3 Library of Congress xxxxxxxxxxx Printed in the United States of America 14 13 12 11 10 9 8 7 6 5 4 3 2 1 405 ii

User Qualifications a thorough knowledge of the procedures and cautions presented in this Manual .

The Direct Observation Form (DOF ) is designed for rating observations of 6-11-year-old children in school classrooms, at recess, and in other group settings. Observers should have some knowledge of child behavior and development and of the methodology of behavioral assessment. Observers may be paraprofessionals, such as teachers’ aides, undergraduate or graduate students, and research assistants, as well as professionals in education, school psychology, clinical psychology, and related disciplines. Paraprofessionals and students should use the DOF under the supervision of a qualified professional who has knowledge of the theory and methodology of standardized assessment.

Our standards for use are consistent with the Standards for Educational and Psychological Testing (1999) prepared and endorsed by the American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME) and with the Code of Fair Testing Practices in Education (2004) prepared by the Joint Committee on Testing Practices. Users are expected to adhere to the ethical principles of their professional organizations, such as the American Psychological Association and National Association of School Psychologists.

To make proper interpretations of the DOF, the data should be scored on the DOF Profile. The ASEBA ADM software provides instructions for computer-scoring the DOF Profile. Interpretation of the DOF Profile usually requires training in standardized assessment commensurate with at least a Master’s degree in psychology, school psychology, social work, special education, counseling, or a com parable field. Trainees, observers, and data processing personnel may also use the computer software to score the DOF Profile under the supervision of a qualified professional. No amount of prior training, however, can substitute for professional maturity and

The DOF is part of the Achenbach System of Em pirically Based Assessment (ASEBA). Users should understand that ASEBA instruments are designed to provide standardized descriptions of an individual’s functioning. The DOF should not be the sole basis for making diagnoses or other important decisions about children and adolescents. No scores on the DOF scales should be automatically equated with a particular diagnosis or disorder. Instead, the responsible user will compare data obtained from the DOF with data from other sources, such as parent reports, teacher reports, child interviews, and observations during test sessions.

iii

Preface The Direct Observation Form (DOF) is part of the Achenbach System of Empirically Based Assessment (ASEBA). This Manual provides basic information needed for understanding and using the DOF. It also provides instructions for co mpleting and scoring the DOF and guidelines for training DOF observers, plus information on development of the DOF, research on reliability and validity, and practical applications with case illustrations of how to integrate DOF results with other assessment information. The DOF can be used to rate and score multiple 10-minute observations of children’s behavioral and emotional problems in school classrooms, at recess, and in other group settings. The DOF includes 89 problem items to be rated on a 4-point scale, plus on-task ratings for each 10-minute observation session. The DOF Profile comprises empirically based scales and DSMoriented scales normed separately for classroom and recess observations for boys and girls ages 6 to 11.

Arnold, Rachel Bérubé, Sarah Cochran, Levent Dumenci, Anne Ellis, Patricia Fletcher, Masha Ivanova, David Jacobowitz, Ramani Sunderaju, and Dan Walter. We are also grateful to the many people who assisted in our data collection and data management, including Lori Turner at the University of Vermont Research Center for Children, Youth, and Families (RCCYF); Ricardo Eiraldi, Thomas Power, and the staff of the Children’s Hospital of Philadelphia; Kevin Antshel, Michael Gordon, and the staff of the Department of Psychiatry at SUNY Upstate Medical University; and Robert Volpe of Northeastern University, who served as a Postdoctoral Fellow at the RCCYF. We are also grateful to the many psychology and school psychology graduate students who acted as observers, as well as the children, families, and school staff who cooperated in our research. We have ap pre ciated the advice of our collea gue s James Hudziak, Cynthia LaRiviere, Leslie Rescorla, James Tallmadge, and Robert Volpe regarding our observational procedures. We are also grateful to the University of Vermont Research Center for Children, Youth, and Families, Spencer Foundation, W. T. Grant Foundation, National Institute of Child Health and Human Development, National Institute on Disability and Rehabilitation Research (U.S. Department of Education), and National Institute of Mental Health for support of research that has contributed to this effort.

In developing the DOF over more than 20 years, we have benefited from the help and advice of many colleagues. For their assistance with this Man ual , we are particularly grateful to Janet

iv

Reader’s Guide I.

Introductory Material Needed by Most Readers A. Introduction and Rationale for the Direct Observation Form (DOF)........... Chapter 1 B. Using the DOF and Rating the DOF Items .................................................. Chapter 2 C. Computer-Scored DOF Profile ....................................................................Chapter 3 D. Training DOF Observers and Conducting School Observations ................. Chapter 4 E. Practical Applications and Case Examples .................................................. Chapter 5

II.

Constructing the DOF and DOF Profile.......................................................... Chapter 6

III.

Statistical Data on Reliability and Validity A. Reliability of the DOF .............................................. ....................................Chapter 7 B. Validity of the DOF ................................................... ..................................Chapter 8

IV.

Answers to Frequently Asked Questions ......................................................... Chapter 9

V.

Mean DOF Scale Scores for Normative Samples of Boys & Girls Ages 6-11 .................................................................................. Appendix A

VI.

Mean DOF Scale Scores for Match Referred Children and Nonreferred Controls Boys 6-11 and Girls 6-11 .................................. Appendix B

VII.

Pearson Correlations Among Raw Scores for DOF Scales .........................Appendix C

VIII. Items Comprising the 2009 DOF and the 1986 DOF................................... Appendix D

v

Contents 1. Introduction and Rationale for the Direct Observation Form (DOF) ....................................................... 1 ADVANTAGES OF DIRECT OBSERVATIONS ..................................................................... ...................... 1 MULTIAXIAL ASSESSMENT ............................................................................................. ......................... 2 STRUCTURE OF THIS MANUAL ............................................................................................................... 3 SUMMARY .................................................................................................................................................... 4 2. Using the DOF and Rating the DOF Items ................................................................................................... 5 COMPLETING PAGE 1 INFORMATION ................................................................... .................................. 5 WRITING OBSERVATION NOTES .............................................. .............................................................. 12 RATING ON-TASK BEHAVIOR ................................................................................................................ . 12 RATING DOF PROBLEM ITEMS .............................................................................................................. 13 GUIDELINES FOR RATING SPECIFIC DOF PROBLEM ITEMS .............................................. ............. 15 SUMMARY .................................................................................................................................................. 22 3. Computer-Scored DOF Profile ..................................................................................................................... 23 DOF PROFILE FOR CLASSROOM OBSERVATIONS ..................................................................... ........ 23 DOF PROFILE FOR RECESS OBSERVATIONS ............................................... ........................ ................ 33 SUMMARY .................................................................................................................................................. 38 4. Training DOF Observers and Conducting School Observations .............................................................. 41 TRAINING DOF OBSERVERS .................................................................................................................. . 41 GUIDELINES FOR OBSERVATIONS IN SCHOOLS ............................................. ......................... .......... 42 ASSESSING INTER-OBSERVER AGREEMENT ..................................................................................... 46 ASSESSING INTER-RATER RELIABILITY ............................................... ........................ ...................... 51 SUMMARY .................................................................................................................................................. 55 5. Practical Applications and Case Examples ................................................................................................. 56 SEQUENCE FOR USING THE DOF AND OTHER ASEBA FORMS ...................................................... 56 SCHOOL-BASED ASSESSMENTS................................................. ........................................................... 59 ASSESSMENT OF ADHD .......................................................................................................................... 61 ASSESSMENT OF EMOTIONAL DISTURBANCE ................................................................................. 61 ASSESSMENT OF LEARNING DISABILITIES ....................................................................................... 62 CASE EXAMPLE OF ASSESSMENT OF ADHD........................ ........................ ........................ .............. 65 CASE EXAMPLE OF A SCHOOL-BASED ASSESSMENT OF BEHAVIOR PROBLEMS .................... 67 SUMMARY ........................ ...................................................................... .................................................... 70 6. Constructing the DOF and DOF Profile...................................................................................................... 71 EARLIER VERSIONS OF THE DOF ...................................................................... .................................... 71 PSYCHOMETRIC APPROACH TO THE 2009 DOF ................................................................................. 73 STATISTICAL DERIVATION OF DOF SYNDROMES FOR CLASSROOM OBSERVATIONS ............. 73 LOW FREQUENCY ITEMS RETAINED ON THE DOF ............................................... ........................ .... 79 AGGRESSIVE BEHAVIOR SYNDROME FOR RECESS OBSERVATIONS ........................................... 79 DSM-ORIENTED ATTENTION DEFICIT/HYPERACTIVITY PROBLEMS AND INATTENTION AND HYPERACTIVITY-IMPULSIVITY SUBSCALES ........................ ........................ ........................ 81 NORMATIVE SAMPLE ..................................................................... ......................................................... 82

vi

Contents

vii

ASSIGNING NORMALIZED T SCORES TO RAW SCORES .............................................. ..................... 82 MEAN T SCORES ........................................................................................................................................ 88 NORMAL, BORDERLINE, AND CLINICAL RANGES ........................................................................... 88 SUMMARY .................................................................................................................................................. 89 7. Reliability of the DOF ................................................................................................................................... 91 INTER-RATER RELIABILITY ............................................ ....................................................................... 91 TEST-RETEST RELIABILITY ............................................ ........................................................................ 93 INTERNAL CONSISTENCY ...................................................................................................................... 94 SUMMARY .................................................................................................................................................. 96 8. Validity of the DOF ....................................................................................................................................... 97 CONTENT VALIDITY OF DOF ITEMS ........................ ........................ ........................ ........................ ..... 97 CRITERION-RELATED VALIDITY .............................................. ............................................................. 98 SUMMARY ................................................................................................................................................ 107 9. Answers to Frequently Asked Questions ................................................................................................... 108 FEATURES OF THE DOF ........................ ...................................................................... ........................... 108 APPLICATIONS OF THE DOF .................................................................................................................. 111 RELATIONS TO OTHER ASSESSMENT PROCEDURES ............................................... ....................... 111 RELATIONS TO DSM AND SPECIAL EDUCATION CLASSIFICATIONS ......................... .................. 112 References ......................................................................................................................................................... 114 APPENDIX A: Mean DOF Scale Scores for Normative Samples................................................................ 118 APPENDIX B: Mean DOF Scale Scores for Matched Referred Children and Nonreferred Controls Boys 6-11 .............................................................................................................................................................. 119 APPENDIX B: Mean DOF Scale Scores for Matched Referred Children and Nonreferred Controls Girls 6-11 ............................................................................................................................................................. 120 APPENDIX C: Pearson Correlations among Raw Scores for DOF Scales for Classroom Observations .................................................................................................................................... 121 APPENDIX D: Items Comprising the 2009 DOF and the 1986 DOF ........................................................ 122 Index ................................................................................................................................................................. 125

Chapter 1

Introduction and Rationale for the Direct Observation Form (DOF) The Direct Observation Form (DOF) is a standardized form for rating observations of children’s behavior in classrooms, at recess, and in other group settings. During a 10-minute period, the observer writes a narrative description of the child’s behavior, affect, and interactions in space provided on the DOF. The observer also rates the child for being on-task or off-task for 5 seconds at the end of each 1-minute interval. At the end of the 10minute observation, the observer rates the child on 88 specific problem items using a 0-1-2-3 scale. Item 89 is open-ended for rating other problems not covered by items 1 through 88. Because children’s behavior may vary considerably from one occasion to another, the DOF com puter-scoring program requires at least two observations of the target or “identified” child. Whenever possible, we recommend 3 to 6 separate observations of behavior on at least two different days. We also recommend obtaining separate observations in the morning and afternoon. Observers should complete one DOF for each 10-minute observation. The DOF computer-scoring program will then average ratings across observation sessions. Because the significance of a child’s behavior depends partly on how it may deviate from the behavior of other children, we recommend observing one or two “control” children in the same setting as the identified child. The control children should be the same age and gender of the identified child, but should be located as far as po ssible from the identified child in the group setting. Observers do not need to know the names of the control children. Chapter 2 provides more detailed instructions for observing identified and control children.

and percentiles for five syndrome scales derived from factor analyses of classroom observations, plus a DSM-oriented Attention Deficit/Hyperactivity Problems scale with Inattention and Hyperactivity-Impulsivity subscales, and a Total Problems score. The DSM-oriented scale and subscales include DOF problem items consistent with symptom criteria for Attention Deficit/Hyperactivity Disorder (ADHD), as defined in the Diagnostic and Statistical Manual of Mental Disorders Fourth Edition and Fourth Edition-Text Revision (DSM-IV; DSM-IV-TR; American Psychiatric Association, 1994, 2000). The DOF also has an Aggressive Behavior syndrome scale for scoring observations during recess and in other non-classroom settings. The DOF Profile has separate norms for boys and girls ages 6 to 11. Because of the com plexity of averaging scores across multiple observation sessions, the DOF scales can only be scored by computer. The DOF computer-scoring program also provides raw scores for each of the 89 problem items and a narrative report that summarizes a child’s scores on each of the DOF scales.

ADVANTAGES OF DIRECT OBSERVATIONS

Direct observation of children’s behavior is a classic assessment method used by clinical and school psychologists (Sattler & Hoge, 2006; Shapiro & Heick, 2004; Wilson & Reschly, 1996). Numerous coding systems have been developed for scoring direct observations of children’s behavior in classrooms (Volpe, DiPerna, Hintze, & Shapiro, 2005) and playground settings (Leff & Lakin, 2005). Systematic direct observations share the following characteristics: (a) their goal is to measure specific target behaviors; (b) the target beThe DOF Profile provides raw scores, T scores haviors are defined in a manner that makes them readily observable with a minimum of inference; (c) the observations are conducted according to 1 standardized procedures; (d) the times and places for observations are specified; and (e) the obser-

2

1. Introduction and Rationale

quantified and/or summarized in a standardized manner that does not vary from one observer to another (Volpe et al., 2005). Many systems for coding observations focus on a limited set of target behaviors (e.g., academic engaged time, out-of-seat, physical aggression, verbal aggression) and rely on continuous recording or time sampling methods. Continuous recording methods count the number of times a behavior (or event) occurs within a given period or record the duration of time in which the behavior (or event) was observed. Continuous recording is most effective when behaviors have discreet beginnings and ends, low to moderate rates of occurrence, and are present only briefly. Time sampling records the presence or absence of target behaviors within short specified time intervals. Time sampling is useful when multiple simultaneous target behaviors hinder continuous recording or when samples of behavior are observed across different settings. The DOF, by contrast, is designed for rating direct observations of multiple specific behaviors over a specific interval (10 minutes). The observer writes a narrative running log of observations over the 10-minute period, while also rating the child as being on-task or off-task during the last 5 seconds of each 1-minute interval. At the end of the 10-minute period, the observer rates the child on each of 89 DOF problem items. The DOF has the following advantages: (a) it provides a structured and efficient method for rating observations of a broad range of specific types of problems; (b) individual problem items are grouped into empirically based syndrome scales, a DSM-oriented ADHP scale and subscales, and Total Problems; (c) norms provide a standard for judging the severity of problems by comparing an individual’s DOF scores to large samples of nonreferred children of the same gender and age range; and (d) scores from DOFs for large samples can be tested for reliability and validity as done for other standardized rating scales.

MULTIAXIAL ASSESSMENT

No one assessment method should serve as the sole basis for evaluating children’s functioning or for making important decisions about children. Instead, responsible evaluators will compare data obtained from one source or method with data obtained from other sources. We use the term, “multiaxial assessment” to describe the process of gathering and integrating information across multiple data sources. To facilitate multiaxial assessment, we designed the DOF as a component of the Achenbach System of Empirically Based Assessment (ASEBA). The ASEBA comprises an integrated set of rating forms for assessing competencies, adaptive functioning, and problems in easy and cost-effective ways. The ASEBA forms most relevant for use with the DOF are the Child Behavior Checklist for Ages 6 to 18 (CBCL/6-18; Achenbach & Rescorla, 2001), Teacher’s Report Form (TRF; Achenbach & Rescorla, 2001), Youth Self-Report (YSR; Achenbach & Rescorla, 2001), Test Observation Form (TOF; McConaughy & Achenbach, 2004), and the Semistructured Clinical Interview for Children and Adolescents (SCICA; McConaughy & Achenbach, 2001). The ASEBA also includes forms for children 1½ to 5, adults ages 18 to 59, and older adults ages 60 to 90. ASEBA data for ages 6 to 11 can be integrated with standardized test data, medical data, developmental history, and other information obtained from records and interviews, as outlined in Table 1-1. The multiaxial assessment model includes the following five axes: Axis I. Parent Data. Standardized ratings of children’s competencies and problems by parents, using the CBCL/6-18, plus history of the child’s development, problems, competencies, and interests as reported by parents in interviews and questionnaires. Axis II. Teacher Data. Standardized ratings of the child’s school performance and problems by teachers, using the TRF, plus history of the child’s school performance as reported by teachers on re port cards, comments in school records, and interviews.


3

Table 1-1 Examples of Multiaxial Assessment Procedures for Ages 6 to 11 Axis I

Axis II

Axis III

Axis IV

Axis V

Parent Report

Teacher Report

Cognitive Assessment

Physical Assessment

Direct Assessment of Child

CBCL/6-18a

TRF b

TOFc

Height, weight

DOFd

History

School records

Ability tests

Medical exam

SCICAe

Parent interview

Caregiver interview

Achievement tests

Neurological exam

YSRf (for age 11)

Perceptual-motor tests

Self-concept mea-

Language tests

Personality tests

sures

a

CBCL/6-18 = Child Behavior Checklist/6-18 (Achenbach & Rescorla, 2001). TRF = Teacher’s Report Form (Achenbach & Rescorla, 2001). c TOF = Test Observation Form (McConaughy & Achenbach, 2004). d DOF = Direct Observation Form (McConaughy & Achenbach, 2009). e SCICA = Semistructured Clinical Interview for Children and Adolescents (McConaughy & Achenbach, 2001). f YSR = Youth Self-Report (Achenbach & Rescorla, 2001). cal interviews, using the SCICA; standardized self Axis III. Cognitive Assessment . Ability tests, such as the Cognitive Assessment System (CAS; ratings by 11-year-olds, using the YSR; self-con Naglieri & Das, 1997), Stanford-Binet Intelligence cept measures, personality tests, and other meaScales-Fifth Edition (SB5; Roid, 2003), Wechsler sures for assessing behavioral and emotional funcIntelligence Scale for Children-Fourth Edition tioning. (WISC-IV, Wechsler, 2003), Woodcock-Johnson III The model in Table 1-1 provides guidelines for Tests of Cognitive Abilities (WJ III COG; Woodmultiaxial assessment of 6-to -11-year-old children. cock, McGrew, & Mather, 2001), and Kaufman AsHowever, not all sources of data may be relevant sessment Battery for Children (KABC; Kaufman or available for every child. For example, self-rat& Kauf- man, 1983); achievement tests; tests of ings may not be useful for children younger than perceptual-motor skills; and speech and language age 11 and children who cannot reflect on their tests. The TOF can also be used by test examiners own behavior. Parents’ reports are highly relevant, to obtain standardized ratings of the child’s test but may not be available from both parents if the session behavior. child lives with only one parent or a parent surro Axis IV. Physical Assessment . Height and gate. Teachers’ reports are usually relevant for weight, physical and/or neurological abnormalischool children if one or more teachers are availties and disabilities, medical and medication hisable to provide them. Standardized ratings of betory. havioral and emotional characteristics observed during testing can add important information about Axis V. Direct Assessment of the Child . Direct a child’s reactions to structured assessment and can observations in group settings, using the DOF; clinihelp examiners judge the validity of test scores. b

4


Direct observations in classrooms or other group settings can be compared with parent and teacher reports and with test session observations. All relevant information from the five axes should be integrated into cohesive formulations of children’s cognitive and behavioral/emotional functioning in order to meet their needs.

STRUCTURE OF THIS MANUAL This Manual provides information for using and scoring the DOF, plus details of its development, standardization, and psychometric properties. User qualifications are presented on Page iii. In this chapter, we discussed our rationale for developing the DOF within the context of a multiaxial assessment model. Chapter 2 discusses how to use the DOF, including how to record observations and rate the DOF problem items. Chapter 3 describes the computer-scored DOF Profile and its narrative re port. Chapter 4 provides guidelines for conducting school observations and training DOF observers. Chapter 5 discusses practical applications of the DOF for use in schools and mental health assessments. Case examples illustrate how the DOF can be used to assess children’s problems and how to integrate DOF results with data from other sources. The remaining chapters present technical details of our research on the DOF and the DOF Pro-

file. Chapter 6 provides background on earlier versions of the DOF, development of the DOF item set, statistical analyses to derive the five DOF syndrome scales for classroom observations and the Aggressive Behavior syndrome scale for recess observations, development of the DOF DSM-oriented Attention Deficit/Hyperactivity Problems scale and its Inattention and Hyperactivity-Impulsivity subscales, assignment of T scores to raw scores, and borderline and clinical cutpoints for the DOF problem scales and On-task. Chapter 7 presents data on reliability, while Chapter 8 presents data on validity of the DOF. Chapter 9 answers frequently asked questions about the DOF and the general approach we have used to deve lop the DOF and its scoring profile. Appendix A presents mean T scores and raw scores, standard deviations, and standard errors for DOF scale scores for the normative sample. Ap pendix B presents mean T scores, raw scores, and standard deviations for matched samples of referred children and nonreferred controls. Appendix C dis plays correlations among the DOF scale scores. Ap pendix D shows the 89 items on the 2009 version of the DOF compared to the 97 items of the 1986 DOF.

SUMMARY We designed the DOF as a standardized form for rating direct observations of children’s behavior in classrooms and other group settings. The DOF Profile for classroom observations displays five empirically based syndrome scales, a DSMoriented Attention Deficit/Hyperactivity Problems scale and Inattention and Hyperactivity-Impulsivity subscales, plus Total Problems scores. The DOF Profile for recess observations has an empirically based Aggressive Behavior syndrome scale and Total Problems score. The DOF scales are scored on norms for boys and girls ages 6 to 11. As part of the ASEBA, the DOF provides data that can be easily compared to data obtained from parents, teachers, youths’ self-ratings, test session observations, and observations during child clinical interviews.

2. Using the DOF and Rating the DOF Items cific problem items (e.g., 17. Tries to get attention of staff or 31. Gets teased ). Rate the child on each DOF problem item according to the following instructions written at the top of Page 3: For each item that describes the child during the 10-minute observation period, circle: 0 = no occurrence; 1 = very slight or ambiguous occurrence; 2 = definite occurrence with mild to moderate intensity/frequency and less than 3 minutes total duration; 3 = definite occurrence with severe intensity, high frequency, or 3 or more minutes total duration.

The intensity of the observed problem and the 3-minute duration are guidelines for choosing ratings of 1, 2, or 3. If it is unclear whether a particular problem occurred or if there was only a slight occurrence, rate the relevant item 1. If a particular problem definitely occurred with mild to moderate intensity or frequency and less than 3 minutes total duration over the course of the 10-minute observation period, rate the relevant item 2. Rate an item 3 if a particular problem occurred with severe intensity, or occurred for 3 or more minutes over the 10-minute observation period, or occurred intermittently for a total of 3 or more minutes throughout the 10-minute observation period. It is not necessary to actually time your observations of each problem. However, it is helpful to have a clock in view so that you can judge whether a problem occurred for at least 3 minutes versus less than 3 minutes. For certain easily observed discreet behaviors (e.g., fidgets, restless, makes odd noises, interrupts), you can make a note each time you observe the behavior to help you judge the frequency of that behavior. Or you can write chit marks next to the initial note of the behavior for each time it occurred during the 10-minute period (e.g., fidgets ////). For certain other discreet behaviors, you can record the amount of time for each instance of their occurrence (e.g., out of seat, 30

15

sec). These notes will help you judge the frequency or intensity of the behavior for rating an item 1, 2, or 3. Other problems (e.g., 11. Confused or seems to be in a fog ; 16. Difficulty following directions) will require your judgment for rating frequency or intensity. Be sure to rate only the one DOF item that most specifically describes a particular observation. For example, several items describe attention problems or hyperactivity, such as 7. Doesn’t concentrate or doesn’t pay attention for long ; 9. Doesn’t sit still, restless, or hyperactive; 13. Fidgets, including with objects; 56. Easily distracted by external stimuli; and 57. Stares blankly. If a child exhibits any such problems during the 10-minute observation period, rate the one item that best fits the actual behavior observed. You may rate more than one item only if the child exhibits more than one different kind of problem, such as difficulty concentrating at certain times, being easily distracted at other times, and being restless. Avoid rating more than one item for the same observation. Figure 2-3 shows the observer’s ratings of Melinda Brandt based on notes for the same 10-minute observation period depicted in Figure 2-3.

GUIDELINES FOR RATING SPECIFIC DOF PROBLEM ITEMS This section provides guidelines to help you choose and rate specific DOF problem items based on our research to develop the DOF. (We have not found it necessary to give guidelines for every item.) You can refer to these guidelines when questions arise during rating. Several guidelines are intended to help you differentiate between similar items. It is not necessary to memorize the guidelines for rating the DOF items. However, you should have the guidelines available when you do your ratings. 1. Acts too young for age . Rate for a child who acts too young or seems immature for his/her chronological age or has mannerisms of a younger child,

6

2. Using the DOF and Rating the DOF Items

Figure 2-1. Page 1 of the Direct Observation Form.


Figure 2-1 (cont.) Page 2 of the Direct Observation Form.

7

8





9

10


tified child and the particular child being observed (identified or control), the observer, and setting. Instructions for completing each field are provided on Page 4 of the DOF. The instructions for each field at the top of Page 1 are shown here in smaller font and discussed in more detail. ID# This space is for an anonymous user-created ID number for the identified child. The ID number is usually assigned by an administrator or other appropriate staff member. The same ID number should be used for control children matched to the identified child.

The space at the top of the DOF is for a userdefined ID number that is unique for each identified child. The same ID number should be assigned to each control child who is linked to the identified child (Control 1, Control 2). The ID number may be created by an administrator or other ap propriate staff member who is coordinating the observations. In some cases, the observer may also assign the ID number if the observer is acting as an independent user (e.g., a school psychologist using the DOF to assess a child). For computerscoring, the ID number will serve as key information for linking an identified child to control children. Identified Child’s Name Write the first, middle (if available), and last name of the identified child (e.g., John Eric Smith). On the DOFs for control children matched to the identified child, write a brief description of the control child (e.g., boy with dark curly hair) and/ or write an abbreviation of the identified child’s name to create a link to the control child (e.g., if the identified child is John Eric Smith, Control 1 might be labeled “JES-C1”).

Whenever possible, write the full name of the identified child. Avoid using initials and writing only the first or last name of the identified child because more than one child may have the same name. However, as discussed in Chapter 4, you

may decide not to record the identified child’s full name on the DOF until after you have left the observation setting so neither the identified child nor peers will see the name of the child being observed. On the DOF for control children, you can write a brief description of the child in the space for the identified child’s name (e.g., boy with dark curly hair; girl with blond hair in front row) to help you identify multiple DOFs for the same control child. Or you can use an abbreviation to link the control child to the identified child. The descriptive information for control children can help to answer questions that may arise when you are trying to identify multiple DOFs for a particular control child linked to the identified child. Child’s Gender Check “Boy” or “Girl” for the gender of the child being observed. Ideally, the gender of the control child should match the gender of the identified child.

Child’s Age On DOFs for the identified child, write age in years. On DOFs for control children, write age of the control child if known, or write age of the identified child as an estimate of the control child’s age, or leave blank.

Administrators, coordinators, or observers should write the age in years of the identified child. Observers do not need to know the names and ages of control children. On DOFs for control children, you can write the age of the identified child as an estimate of the control child’s age or leave this space blank. Child’s Ethnic Group or Race Write the known or apparent ethnic group or race of the child being observed (e.g., White, African American, Asian).

In this space, write the known or apparent ethnic group or race of the child being observed (Identified Child, Control 1, Control 2). You can use your own terminology for ethnic group or race or choose from a list of terms. The DOF computerscoring program provides the following list of

2. Using the DOF and Rating the DOF Items terms for data entry: African American, Asian, Latino/Latina, Native American, Pacific Islander, White (non-Latino), Other. You can also create your own terms for this field for data entry. Observer’s Name Write the observer’s first and last name or initials.

Observation # Write a separate unique number for each 10minute observation for the identified child (e.g., 1, 2, 3, 4, 5, 6) and each 10-minute observation for each control child.

Write a separate unique number for each separate 10-minute observation in sequence for each individual child. For example, if you observe the identified child six times, record observation num bers 1, 2, 3, 4, 5, and 6 for each of the six DOFs in sequence for the identified child. The six observations may span the course of several days. The observation number, “Today’s Date,” and “Time of Day” should be consistent with the sequence o f observations. If you observe one control child (Control Child 1) twice, record observation num bers 1 and 2 for each DOF in sequence for that control child. If you observe a second control child (Control Child 2) twice, record observation num bers 1 and 2 for each DOF in sequence for that child. Grade or Level Write the grade (e.g., Kindergarten, 1st, 4th) or level in school (e.g.,1-2) of the child being observed. Ideally, the grade or level of the control child should match the grade or level of the identified child.

Identified Child’s Birthdate Write the identified child’s birthdate.

The DOF and the DOF computer-scoring program use month-day-year format for birthdate. On DOFs for a control child, write the birthdate of the identified child. In addition to the ID number and identified child’s name, the birthdate will provide another way to link control children to the appro priate identified child.

11

Observation Set Assign a label to identify the set or group of DOFs for the identified child and control children to be computer-scored on the same DOF Profile . This might be a time frame for the set of observations (e.g., Fall 2009) or a specific setting for the observations (e.g., math class, library). The computer-scoring program allows a minimum of 2 and maximum of 18 DOFs as an observation set to be scored on one DOF Profile: 2 to 6 DOFs for the Identified Child, 1 to 6 for Control 1, and 1 to 6 for Control 2. DOFs for control children are optional.

Observation set is a required field for computerscoring. When you enter each DOF into the com puter-scoring program, you must assign a label to identify it as a member of a set of DOFs that will be selected as one group to be scored on the same DOF Profile. As explained in Chapter 3, the com puter-scoring program averages ratings on DOF items across multiple DOFs separately for the identified child and matched control children. You can use any label that is meaningful to you to identify which DOFs will form a set for the averaging process in computer-scoring. Examples are a label for a time frame for the set of observations or a specific setting for the observations. You can use the same observation set label for a minimum of two and maximum of 18 DOFs for computer-scoring. DOFs for control children are optional. There must be at least two DOFs for control children when observations for control children are included in an observation set, as explained in Chapter 3. Observed Child Check one box to indicate whether the observed child for each DOF is the “Identified Child,” “Control Child 1,” or “Control Child 2.”

This is a required field for computer-scoring. Check the box, “Identified Child,” to indicate that the observed child was the selected identified child whose name is recorded on the DOF form. Check the box, “Control Child 1,” for the first control child in the same setting who is to be matched to the identified child. Check the box, “Control Child 2,” for the second control child in the same setting who

12


is to be matched to the identified child. Whenever possible, the gender of the control children should be the same as the gender of the identified child. Time of Day Write the time of the beginning of the 10-minute observation in hours and minutes and a.m. or p.m. (e.g., 9:20 a.m., 12:30 p.m.)

Today’s Date Write the date of the observation.

The DOF and the DOF computer-scoring program use month-day-year format for the date of the observation. Setting Check one box to indicate whether the observation was conducted in the classroom or at recess. If you conduct an observation in a setting other than class or recess, choose the setting option that most closely approximates the activity of children in that particular setting (e.g., lunch = recess; small group instruction = class). You can use the space to write the type of activity for classroom observations (e.g., math, reading, circle group) or recess observations (e.g., inside games, outdoor play).

This is a required field for computer-scoring. Choose only one setting (Class or Recess) for each DOF. The computer-scoring program uses these two fields to determine whether the ratings from that DOF will be scored on a DOF Profile based on norms for classroom observations or norms for recess observations. There is no option for “Other” setting because there are no normative data for scoring observations in settings other than class or recess. You also have the option of recording the type of activity for each DOF for classroom observations (e.g., math, reading, circle group) or recess observations (e.g., inside games, outdoor play).

WRITING OBSERVATION NOTES Use the spaces provided on Page 2 (see Figure 2-1) to write a narrative description of the child’s behavior, affect, and interaction style over the 10minute observation period. You do not have to write

complete sentences. Instead, record brief notes and abbreviations that will help you rate the 89 DOF problem items listed on Page 3. The numbered boxes in the left-hand column on Page 2 demarcate 1-minute intervals for rating on-task, as ex plained in the next section. By scanning the list of DOF problem items before each observation session, you can familiarize yourself with the types of behaviors to describe. When appropriate, note the frequency (e.g., by chit marks), duration (e.g., 20 sec), or intensity of specific problems to help you choose between ratings of 1, 2, or 3 for each problem item. Sometimes, you may want to describe events during the 10minute observation that affect the child’s behavior, such as the teacher’s behavior or behavior of peers. For example, you may observe that a child daydreams or is restless during independent seat work in class, but does not show these problems when the teacher works with him/her directly. Or you may observe that a child is teased or hit by another child, and subsequently teases back or becomes involved in a fight. You may consider these interactions when rating the child’s behavior on relevant DOF problem items. However, you should avoid making inferences about the child’s motivations when rating specific DOF items, as instructed in a later section. Remember that DOF items are to be rated only for behavior observed in the 10minute window for the observation period. The 10minute observation window also applies to any events that might affect the child’s behavior.

RATING ON-TASK BEHAVIOR The left-hand side of Page 2 of the DOF (see Figure 2-1) contains 10 boxes in 2 columns for rating whether the child is on-task (ON TASK) or not on-task (OFF TASK). These boxes represent 5-second intervals at the end of each minute of ob servation. In the last 5 seconds of each 1-minute interval, observe the child’s on-task behavior. If the child’s behavior is on-task during the 5-second interval, draw a line through the box for “ON TASK.” If the child is not on-task, draw a line


13

through the box for “OFF TASK.”

tions.

Consider a child to be on-task if he/she is doing what is expected in that situation (e.g., listening to directions, reading a book, working on an assigned task at his/her desk, listening to others in circle time, etc.). The child should be on-task for the ma jority of the 5-second interval. You can use a stopwatch to indicate each 1-minute interval if you wish, but this is not required. Another option is to watch the second hand on a clock or your watch and start each 1-minute on-task observation at a specified time (e.g., when the second hand is on the 11).

Figure 2-2 illustrates an observer’s notes and on-task ratings for the first 10-minute observation of 8-year-old Melinda Brandt (not her real name), whose computer-scored DOF Profile is presented in Chapter 3. Melinda is also discussed as a case example in Chapter 5. The complete set of observations included four 10-minute observations of Melinda and two 10-minute observations of each of two control children in the same class.

If the child is not on-task for the majority of the 5-second interval, rate the child as off- task. The following are examples of when a child is “offtask”: The child does something that requires the teacher to redirect him/her to get back “on-task.” The child is doodling or drawing or playing with a toy or other object when he/she is supposed to be listening to the teacher or working on an assignment. The child is looking around the room or is not looking at the teacher or someone else who is speaking to him/her or to the whole class. The child is poking another student, talking to another student, or clowning when he/she is sup posed to be listening or working quietly. At the end of the 10-minute observation period, count the number of intervals you rated the child as off-task and write the sum in the box for SUM OFF TASK. Count the number of intervals you rated the child as on-task and write the sum in the box for SUM ON TASK. The total number of intervals rated for SUM OFF TASK + SUM ON TASK should not exceed 10. The computer-scoring program averages on-task ratings across multiple DOFs separately for the identified child and for controls. The total number of intervals for ontask and off-task on a single DOF must be > 8 for computer-scoring. On-task ratings are only scored for classroom observations, not recess observa-

RATING DOF PROBLEM ITEMS Immediately after completing each 10-minute observation, rate the child on the 89 DOF problem items listed on Page 3. Be sure to complete your ratings of DOF problem items before you start a nother 10-minute observation. Problem behaviors do not have to attract the attention of the school staff in order to be rated as present. Equally im portant, your ratings of problem items should not depend on your ratings of whether the child was “on-task” or “off-task.” For example, a child may be considered on-task while working on an assignment, but still be restless, or fidget, or look unhappy. Some problems, such as 7. Doesn’t concentrate or pay attention for long , can suggest the child is off-task. However, it is possible that a child could have problems concentrating during parts of the observation period, but then be on-task during the last 5 seconds of a 1-minute interval. To rate the DOF problem items, choose the one item that specifically reflects each behavior actually observed during the 10-minute observation period . Review your notes on Page 2 to help remember your observations. As you read the DOF problem items, you may also remember some behaviors that may not have been described in your notes. You can rate such items even if you did not write the specific behavior in your notes. (As you become more familiar with the DOF problem items, your observation notes should become more closely aligned to your item ratings.) You may also consider interactions with teachers and peers during the 10-minute observation period to rate spe-

14


Figure 2-2. Observer’s notes and on-task ratings for the first 10-minute observation of Melinda Brandt.

2009_DOF_Manual[1]

Recommend Documents