Evaluating & Assessing IPAS Projects: Stories from the Field D. Christopher Brooks, Ph.D. Senior Research Fellow ECAR
Laura Damm Student Success Coordinator East Mississippi Community College
Lois Smith Vice President, Organizational Growth, Development & Effectiveness Stanly Community College 2015 NERCOM P Annual Conference, 31 March 2015
Overview What is IPAS? ECAR IPAS Projects Methodological Issues Case Studies
Integrated Planning & Advising Services
ECAR IPAS Research Hub http://tinyurl.com/k95hr7t
IPAS Evaluation & Assessment Guide IPAS • Why the evaluation and assessment to IPAS projects is important
• Issues and approaches to designing an evaluation and assessment protocol • Issues and approaches related to measuring and analyzing outcomes data • Identifying good practices in assessing and evaluating IPAS projects IPAS evaluation and assessments lead to the • How IPAS identification of actionable outcomes
Evaluation versus Assessment Evaluation
Assessment
Question
How do we know the efforts are or have been worth it?
How do we know if it works or worked?
Objects
Worth or merit of a Impact of a project project
Aspects
Procedurally-focused Outcomes-focused Judgmental Diagnostic Contextual Empirical
Evaluation versus Assessment: Timing
Evaluation & Assessment: Levels of Analysis
Assessment Designs
Assessment Designs
Case Studies
East Mississippi Community College 17-18 December 2015
Stanly Community College 29-30 January 2015
Montgomery County Community College 2 February 2015
Colorado State University 5 February 2015
East Mississippi Community College Expand the Pride Laura Damm Student Success Coordinator
[email protected]
March 31, 2015
Campuses Scooba- The main campus (est.1927) is residential, host to athletics, band, and chorus in addition to academic and limited technical programs. 1,126 students Mayhew- The most populous, commuter campus (est. 1968) houses GED and Workforce training services in addition to academic and technical programs. 3,673 students Four extension centers offer unique technical programs. Online course offerings began in 2000.
Demographics EMCC enrolls about 6,900 students annually Ethnicity is 49% White, 47% Black, 4% Other Female (58%) Attends full time (57%) Average age is 23.5 Most are commuters
Technology solutions 1. DropGuard by Smartevals Early Alert system driven by instructor input Replaced homegrown process initiated 2009 and stalled in 2010-11. 2010-11. Went live fall 2012. 2. Student Planning by Ellucian Education planning tool, asynchronous interaction with advisor, advisor, capability to register with one click. Went live fall 2013. 1.
Project Goals 1. Increase retention of new cohort 10% 2. Early Alert used by 100% FT instructors; 75% adjuncts in year 1 and 85% in year 2 3. Student Planning used with 25% new cohort in year 1 and 50% in year 2 4. New cohort demonstrates 5% improvement in retention, persistence, and graduation/transfer 1.
Usage of Student Planning by Students in FTFT Fall 2013 cohort
This is a viable tool that allows a second party to reach out to my students via email contact and phone calls. Through this tool, students are encouraged to meet with tutors, use technology more effectively, and have a one-on-one question and answer session that addresses EMCC policy and procedures. -S. Frey, faculty
Completion Cohort: First time, full time students enrolled at 10th day audit. Q: Of those who started, how many completed?
Retention/Course Completion
Re-enrolled Cohort: First time, full time students enrolled at 10th day audit. Q: Of those who started, how many re-enrolled?
Retention/Re-enrolled
Warnings Students who received a warning for poor test, missing assignment, online absence, absent from class, and missing English essay.
Warned with Early Alert System
Intervened Intervened: Defined as on ground tutoring due to inability to track individual students in NetTutor
Intervened with on ground tutoring
Warned and Finished How many students in the cohort were warned and finished the term?
Warned and Finished
Tutored and Finished How many students in the cohort were tutored and finished the semester?
Tutored and Finished
Lions’ Brew opened Sept 2014. Usage of tutoring services increased 155%.
Evaluating and Evaluating and Assessin Assessing g IPAS IPAS Projects: Stories from the Field Lois Smith VP Organizational Growth, Development, and Effectiveness Stanly Community College North Carolina March 31, 2015
Stanly Community College !
A little about us !
!
!
Small, primarily rural, community college in south central North Carolina One of 58 community colleges in NC SCC serves Stanly County
Stanly Community College !
!
!
A member college of Achieving the Dream and of the Completion by Design NC cohort Curriculum student headcount of approximately 2800 Smallest college in the Gates Foundation IPAS Grant Program
Stanly Community College
!
IPAS Project at Stanly •
The technology selected was Student Success Plan (SSP)
•
SSP is open source software that was integrated with our Moodle LMS and our Ellucian SIS
•
Used two partners to assist with implementation - Unicon and Gateway to College
•
Users throughout the institution are utilizing SSP for: Early Alert, Journaling, and My Academic Plan (MAP) for academic planning
Stanly Community College !
Goals and Outcomes Upon completion of the IPAS adoption, SCC: !
!
!
will report improvement in cross department collaboration for the benefit of student success and completion will be able to offer students a holistic approach to advising and intervention strategies with the goal of student retention will be able to compare course availability to projected student needs via MAPs
Stanly Community College !
Goals and Outcomes Upon completion of the IPAS adoption, SCC: !
!
!
will be better able to carry out student risk assessment and take corrective actions with the goal of student assistance and retention students will be better able to access support services students will better understand their academic pathways
Stanly Community College !
Assessment Strategy !
Long Range and Short Range Measures
!
Few Examples: !
!
!
Indirect measures – student surveys (those who received early alerts or received a MAP) Usage reports within SSP such as # of completed MAPs or # of early alerts generated Institutional data such as Student Persistence, Success Rates, and Course Completion Rates
Stanly Community College !
Assessment Strategy - Example !
Holistic approach to advising and interventions measured by tracking early alert activity and student persistence
Stanly Community College !
Assessment Strategy - Example !
Student Persistence rates, are so far, far, remaining steady when compared back to baseline
Spring 2013 to Fall 2013: 52.8% (Baseline) Spring 2014 to Fall 2014: 52.9%
Stanly Community College !
Assessment Strategy – Example !
!
Overall goal of improving student success and course completion measured by: Student Success Rates Fall 2013 (Baseline) = 75% Fall 2014 (Post-SSP) = 78%
!
Student Course Completion Rates Fall 2013 (Baseline) = 83% Fall 2014 (Post-SSP) = 86%
Stanly Community College !
!
The SCC IPAS IPAS story in a nutshell. Assessment to date is showing the IPAS IPAS project and SSP implementation has been beneficial for our college and our students.
Questions?
!"#$ &'()*(+,- (-. #//0//10-2 3*4.0
D. Christopher Brooks Coming: May 2015 to http://tinyurl.com/k95hr7t
Thank you! D. Christopher Brooks
[email protected] @dcbphd Laura Damm
[email protected] Lois Smith
[email protected]
