Summer Melt Predictive Model

BACKGROUND

“Summer melt” is the phenomenon occurring widely throughout the United States, in which high school graduates intending to attend college change their plans and fail to enroll by the time the fall semester begins. This trend affects an estimated 10-20% of graduates nationwide (U.S. Department of Education) and ultimately limits the economic growth of many cities. EdjAnalytics and a K-12 public school system in Kentucky partnered to address the issue of summer melt within their district. The vision of the partnership was to predict which students were most likely to experience summer melt, to identify the reasons behind those predictions and to combat the issue of summer melt within the community.

APPROACH

The school system provided data from all four years of high school students graduating between 2014 and 2016. In total, 15,500 students were analyzed. Of the approximately 14,000 students who indicated an intent to attend college, 33% melted. Data analyzed included demographics, socio-economic factors associated with home zip codes, attendance and suspension records, course grades and standardized test scores. College intent was determined from the Senior Transition Survey and college enrollment was defined using National Student Clearinghouse records.

RESULTS

Using advanced analytics, Edj demonstrated that academic indicators are most predictive of whether a student will matriculate at college. ACT scores were shown to have the strongest relationship to college follow-through. The model showed that students who have a composite score above 20 are likely to attend college, as originally indicated. However, students who have a composite score of 15 or lower or who did not take the ACT are highly likely to melt.

Summer Melt ACT Score

While ACT scores are highly predictive indicators of future behavior, they are primarily taken late in the Junior year which precludes educators from intervening earlier in the student’s career. Edj was challenged to find another way to flag at-risk students earlier in their high school careers.

Course grades granted throughout high school offer an earlier look at potential outcomes. The letter grades received and the courses for which they were received proved be a clear predictor of summer melt, comparable in strength to ACT scores. Consider an example from Health Education, a mandatory class for all high school students in Kentucky. Students who earned an A- or higher were unlikely to melt. Because Health is taken during Freshman year, Health grades can be used as part of an early warning system to identify students at risk for future melt.

For courses with levels such as Advanced or Honors, Edj found that students in higher-level classes receiving anything above a D were still under the average melt rate of 33%. Those who earned As and Bs in Advanced or Honor courses were at considerably lower risk of melting. In contrast, Edj found that the majority of students who are in the Regular classes are at or above the average melt percentage, even if they are earning As.

summer melt algebra

In addition to examining test scores and grades, Edj built models that looked at the predictability of using only demographic information, only attendance and suspension data and only socio-economic data. None of those models were as strong as the models built using individual academic indicators.

The area under the ROC curve (AUC) was used to judge the predictive quality of each model. The closer the AUC is to 1, the more accurate the model is at predicting outcomes. The strongest predictive model was the model built using all of the available variables, generating an AUC of 0.81, which is very strong. However, the models that looked at only academic variables were also highly predictive: the ACT Subject Score model’s AUC was 0.77, followed by the ACT Composite Score, the Course Grades and the QualityCore models, each generating an AUC of about 0.75. In comparison, models using only demographic data performed at a 0.57 AUC, which is only slightly better than a randomized guess.

in-sample auc

To summarize the findings, Edj concluded that course rigor and academic success are highly predictive of whether a student decides to follow through with their plans to attend college. While these findings may seem intuitive, what is eye-opening is the diminished role socio-economic status plays and the early warning signs that exist among academic data (for example, a B- grade in Health class) that is an attainable opportunity for educators.

NEXT STEPS

Edj and the school system plan to continue their work together to improve student outcomes. Edj is investigating additional partnership opportunities with additional school systems in Kentucky to address the broader issue of college readiness and build a statewide K-12 success platform. This tool would look at all of the academic and social variables contributing to success along the comprehensive K-12 career path, with a goal of predicting which students are at risk of missing key milestones and measuring the likelihood of potential outcomes. Outcomes of interest include identifying those students unlikely to achieve third grade reading level, to gain proficiency in math before fifth grade, to graduate from high school and to continue on to college.