Measuring the Effectiveness of College & Career Readiness Interventions CAROLYN BERGER, PH.D. NOVA SOUTHEASTERN UNIVERSITY
[email protected]
Overview
Importance of CCR data
Deciding what data to collect
Established CCR assessments
Developing an action plan
Presenting data to stakeholders
Access full presentation at: http://bit.ly/EBSC-2016-Berger
What do you think??
School Counselors must collect all types of data on their counseling program.
Do you think collecting CCR data is especially important?
Why or why not?
Importance of CCR data
National attention around CCR issues (Reach Higher, North Star Goal, etc.)
The Center on Education and the Workforce reported that:
By 2020, 65% of jobs in the nation will require some form of postsecondary education.
At our current rate of educational attainment, the U.S. will be short 5 million workers for jobs that require postsecondary credentials by 2020.
School counselors need to demonstrate they can help work towards national and state goals in CCR areas. It’s our time to shine!
DATA is essential to demonstrate our role in this important initiative.
Deciding what data to collect
There are a lot of different options for CCR data
See following slides for data that corresponds with the 8 NOSCA Components of CCR
Pair & Share
How can school counselors figure out which CCR data their principal or school district would most want to see improve?
Deciding what data to collect
Look at school improvement plan (i.e., school goals)
Collect all 3 types of data: Process
(# students)
Perception
Outcome
(e.g., surveys)
(e.g., test scores, grades, attendance, etc.)
Think about what stakeholders want to see improved
Outcome data is preferred but perception can be used to support
Deciding what data to collect NOSCA includes a wide variety of PROCESS DATA & OUTCOME DATA
Process
Participation in extracurriculars & enrichment activities
Outcome
Attendance
Discipline
Promotion
GPA
# students in leadership positions
Participation in career/interest assessments
Participation in financial aid talks
Dropout
Student enrollment in advanced courses
Reading level
Standardized tests
Enrollment in AP
Deciding what data to collect
Perception data is not included in NOSCA’s list
Several types of assessments that can gather perception data will be addressed
Perception data isn’t usually as preferred by administrators as outcome data
However, perception data can give a lot of useful information to best meet students’ CCR needs
Why is perception data so useful?
Do students perceive that the counseling interventions are helpful? Do students perceive that their cognitive skills, attitudes, or knowledge levels have been positively impacted as a result of a counseling intervention? Perception data is useful to demonstrate:
Knowledge of CCR skills (e.g., how to apply, requirements to pursue postsecondary education, etc.)
Confidence levels
Career maturity
Established CCR Perception Assessments (click on links)
Career Beliefs Inventory (CBI)- 96 items, grades 8-12, 5 categories, (e.g. Factors that influence my decisions)
Career Decision Making Self-Efficacy Scale (CDSE)- long or short forms, high school, 5 broad areas (e.g., self-appraisal)
Career Thoughts Inventory (CTI)- 48 items, 3 scales (e.g., commitment anxiety), high school
College Going Self-Efficacy Scale (CGSES)- 30 items, measures beliefs about college attendance and persistence, MS and HS
Career Development Inventory (CDI)- 80 items, four aspects of career maturity (e.g., career planning), 1 hour, high school
Career Maturity Inventory (CMI)- 24 items (agree or disagree), 6th-12th grade, measures students’ maturity towards making career decisions
Career Beliefs Inventory
96 items
grades 8-12
Five Scales: 1.
My Current Career Situation
2.
What Seems Necessary for My Happiness
3.
Factors That Influence My Decisions
4.
Changes I Am Willing to Make
5.
Effort I Am Willing to Initiate
Career Beliefs Inventory
Manual = $50; Reproducible PDF = $2 per assessment; Online survey = $2 per assessment
Career Decision Making Self-Efficacy Scale
long or short forms (50 vs. 25 items)
high school
5 broad areas:
Self-Appraisal: The ability to accurately appraise one's own abilities, interests, and values
Occupational Information: The ability to locate sources of information about college majors and occupations
Goal Selection: The ability to match one's own characteristics to the demands and rewards of careers
Planning: Knowing how to implement an educational or career choice,
Problem Solving: Being able to figure out alternative plans or coping strategies when plans do not go as intended.
Career Decision-Making Self Efficacy Scale
Manual = $50; Reproducible PDF = $2 per assessment; Online survey = $2.40 per assessment
Career Thoughts Inventory
Assist in career problem solving and decision making in adults, college students, and (upper) high school students
$59 manual
$74/25 text booklets
Likert scale: Strongly Agree- Agree- Disagree- Strongly Disagree
48 Items; Three scales: 1.
Decision Making Confusion
2.
Commitment Anxiety
3.
External Conflict
CTI Sample Items
College-Going Self-Efficacy Scale
Two scales: Attendance Scale & Persistence Scale
30 Items
Links:
Free access to CGSES
Information about reliability & validity
CGSES Sample Items
College-Going Self-Efficacy Scale
For attendance beliefs, school counselors can examine concerns about finances, abilities, family, decision-making, and general feelings about attending college.
For persistence beliefs, school counselors can focus on ability, finances, family responsibilities, life skills, and general feelings about continuing in college.
Career Development Inventory
Free on Vocopher.com (register for account online); includes a User Manual and research on validity & reliability
80 items
Four Subscales: 1.
Career Planning
2.
Career Exploration
3.
World of Work knowledge
4.
Decision Making
CDI
The Career Development Inventory was created to:
Measure career maturity
Address career concerns in adolescence
CDI Sample Items
Career Maturity Inventory
24 Items
Free on Vocopher.com (register for account online)
Surveys attitudes which are important in making career decisions
Asks about attitudes and feelings toward making a career choice and entering the world of work
Helps counselors see how mature their students are and identify specific areas of concern
CMI Sample Items ITEM
Response
Once you choose a job, you can't choose another one.
AGREE or DISAGREE
In order to choose a job, you need to know what kind of person you are.
AGREE or DISAGREE
I plan to follow the line of work my parents suggest.
AGREE or DISAGREE
I guess everyone has to go to work sooner than later, but I don't look forward to it.
AGREE or DISAGREE
You can do any kind of work you want to as long as you try hard.
AGREE or DISAGREE
CMI Sample results
Others??
Structured Career Development Interview (Lapan, 2000)
Work Adjustment Inventory (Gilliam, 1994)
Occupational Self-Efficacy Scale (Schyns & von Collani, 2002)
More?
Recommendations for Choosing Assessments
Take them yourself in advance
Make sure they measure what you intend to measure: what is the main purpose of your intervention?
Pilot with a few students of the same age as your intended population
What if I can’t find the assessment I need? Consult
with counselor educators who may be able to assist you
You
may wish to consider Instrument development but be aware that it is a multistep process
Important
to establish reliability and validity
Developing an Action Plan
Pair up with a fellow attendee and discuss your own plans for measuring CCR data at your school
Which types of CCR data do you think your administration and other stakeholders want to see?
What types of data will you collect?
What assessments do you want to research further and perhaps implement?
Presenting Data to Stakeholders
Closely tie results to school goals for CCR
Go back to the question, “what does my administration want to see improved?”
If you cannot demonstrate clear results using outcome data, utilize perception data to show how the intervention helped.
Always discuss next steps when presenting data
If it works, discuss plan to do more of it.
If it didn’t work that great, what will you change?
Presenting Data to Stakeholders
Make sure you know how to CLEARLY and SUCCINCTLY present your data
Charts should be easy to decipher- show to a few “laypeople” to make sure they can read them easily
Use a clear model for presenting data – e.g., flashlight approach (Hatch), MEASURE/SPARC (Stone & Dahir)
References
Bowers, J. & Hatch, C. (2012) – Third Edition. The ASCA National Model - A framework for school counseling programs. Alexandria, VA: The American School Counselor Association.* ISBN-13: 978-1-929289-32-5
The College Board (2010). The College Board National Office for School Counselor Advocacy: Eight components of college and career readiness counseling. Retrieved from https://securemedia.collegeboard.org/digitalServices/pdf/nosca/11b_4416_8_Components_WEB_111107.pd f
Curry, J. & Milsom, A. (2014). Career Counseling in P-12 Schools. New York, NY: Springer Publishing.
Hatch, T. (2014). The Use of Data in School Counseling. Thousand Oaks, CA: Corwin.
Stone, C. B. & Dahir, C. A. (2011). – Third Edition. School counselor accountability: A MEASURE of student success. Upper Saddle River, NJ: Prentice Hall.
Thank you for coming!!
Full presentation accessible at: http://bit.ly/EBSC-2016-Berger
Feel free to contact Carolyn Berger at
[email protected]
