Evaluating College Support
A study of the effects of targeted college aid offered by the Susan Thompson Buffett Foundation.
The Susan Thompson Buffett Foundation (STBF) offers scholarships to Nebraska high school graduates who attend the state’s public colleges and universities.
STBF is one of the largest private funders of college scholarships in the country. Awards cover the full cost of tuition and fees (plus $500 toward books) to attend college at any Nebraska state university or college. The scholarships can provide more than $70,000 toward five years of study.
Many of the scholarship winners who attend one of the three University of Nebraska campuses (in Lincoln, Kearney, or Omaha) also participate in Learning Communities, which provide academic and personal support in addition to social programming. The research team is tracking the progress of NU students who do and do not participate in Learning Communities and will share results that isolate the effects of Learning Communities in future research.
STBF provides over $40 million in aid every year to more than 4,000 students.
This study includes applicants to the STBF scholarship program between 2012 and 2016. In 2012, more than 2,000 high school students in Nebraska applied for a STBF scholarship.
In response to outreach and advertising, applications grew to almost 4,000 annually by 2016. From 2012 through 2016, roughly 16,500 students applied for a STBF scholarship.
To be eligible for the scholarship, applicants must have had an unweighted high school grade point average of 2.5 or higher and documented financial need (an expected family contribution below $15,000 in 2012 and $10,000 in 2013 through 2016).
Of the 16,500 applicants, 11,000 were eligible for a STBF scholarship. We grouped these eligible applicants by their first-choice campus. The two-year group (about 15% of eligible applicants) includes applicants who said that, if admitted, they would attend a two-year college, while the four-year group (85% of eligible applicants) includes applicants who targeted a four-year college.
Applicants were scored based on academic achievement, financial need, a personal essay and letters of recommendation.
The highest scoring applicants were awarded aid. The lowest scoring applicants did not receive an award.
Our study focuses on the middle group, applicants who were all equally deserving of an award. Between 2012 and 2016, nearly three-quarters of eligible applicants were in this middle group.
STBF is interested in understanding the impact of the scholarship on student outcomes such as college enrollment, persistence, degree completion, and college debt.
How should we evaluate a scholarship program? One might simply compare applicants who receive scholarships with those who don't, but this isn’t the best approach. STBF aid is awarded partly on merit, like many other programs, such as state college aid for students with a strong high school record. High-performing students are likely to be awarded a scholarship, while low-performing students are not. The students who receive scholarships would likely complete college at a higher rate even without aid.
To solve this problem, the research team deployed a partially randomized design that focuses on applicants in the middle of the pool. As the middle group of applicants were deemed all equally qualified for an award, STBF randomly awarded scholarships to up to half of this group in each year. Randomization is fair in this context because the number of applicants with scores in the middle range exceeds the number of awards available.
Random assignment ensures that, on average, applicants in the middle group who did and did not receive aid were similar before STBF awarded scholarships.
Statistical comparisons show that the applicants in the middle group who were and were not offered STBF awards are indeed similar. In particular, they have similar:
- Demographic characteristics, such as gender and race;
- High school grade point averages and composite ACT scores;
- Family incomes and parental education backgrounds.
Since these two groups were nearly identical before the award and the awards were assigned randomly, any difference in outcomes reflects the causal effect of the STBF scholarship.
These results are part of an ongoing research study. Our current analysis looks at enrollment and college completion through spring 2019. Later analyses will include more data as subsequent cohorts of students in the study reach the date of expected college completion. The results that follow separate the sample into randomized applicants who targeted four-year schools and randomized applicants who targeted two-year schools.
* Small discrepancies between the point estimates referenced in the text and the graph values are due to rounding. The downloadable data contains the precise estimates.
Scholarships Increase Freshman Financial Aid Packages
The gray numbers above each pair of bars indicate the percent change in each aid category for applicants who receive an award. For example, the graph shows a 32 percent increase in total aid for award winners. The small interval markers overlapping the top of each bar indicate a 95 percent confidence interval. This means that if the STBF scholarship study were repeated 100 times with different random samples of students, each time calculating the 95 percent confidence interval for the new sample, we would expect that 95 percent of those confidence intervals would contain the true population mean.
Susan Thompson Buffett Foundation (STBF) award amounts are calibrated to match the cost of tuition and fees for each student based on their part-time or full-time status, plus $500 for books. Each pair of bars shows the effect of a STBF scholarship on the category listed at the bottom of the graph in the applicant’s freshman year. The two bars to the far left indicate that STBF award winners on average receive $8,158 in aid compared to their peers who receive $0 in aid from the Foundation. Compared with students who did not receive an award, award recipients receive double the grant aid and thus rely less on federal loans and work-study programs.
Scholarships Allow More Students to Finish College Loan-Free
46 percent of award recipients finished their fourth year without any federal loans. By comparison, only 29 percent of those who did not receive an award had no federal loans after four years. STBF awards increased the share of students who had no federal loans in year four by 17 percentage points.
Scholarships Boost Postsecondary Enrollment
In the graph above and the ones that follow, the shaded area indicates a 95 percent confidence interval. This means that if the STBF scholarship study were repeated 100 times with different random samples of students, each time calculating the 95 percent confidence interval for the new sample, we would expect that 95 percent of those confidence intervals would contain the true population mean. In this chart, the shaded area around the line for students who received a scholarship does not overlap with the line for students who were not awarded a scholarship, suggesting that our results are not chance findings.
STBF awards boosted enrollment and cut dropout rates in half after four years. Enrollment among students who did not receive a scholarship fell from 96 percent in students’ first semester to 78 percent in the fall semester of students’ fourth year, a decline of 18 percentage points. Enrollment among students who were awarded scholarships fell by only 13 percentage points over the same period.
STBF awards also increased enrollment for students who targeted a two-year college. More than half of STBF award winners were still in school in the fall, four years after winning the award, compared with less than half of students who did not win an award. Some students who start in a two-year program transfer to a four-year school with continued STBF support.
Scholarships Shift Students with Below-Median ACT Scores to Four-Year Colleges
In the graphs above, each dot represents a cluster of students with similar ACT scores. On the horizontal axis, ACT scores (which range from 1 to 36) are converted to percentile values. Each cluster represents 10 percent of the control or treatment group.
These graphs show the relationship between students’ ACT scores and college enrollment. The first graph shows the share of students enrolled in year one at any postsecondary institution (i.e., a two- or four-year college). Enrollment rates among award recipients and other students are not very different: regardless of ACT score, both groups enroll in college at roughly the same rate.
Initial enrollment in four-year colleges differs by ACT score. Award recipients with lower ACT scores are more likely to enroll in a four-year college than are similar-scoring students who did not receive an award. Conversely, award recipients with high ACT scores enroll in four-year colleges at the same rate regardless of whether they receive an award. STBF awards appear to have shifted students with below-median ACT scores from two-year colleges to four-year colleges.
Scholarships Boost Bachelor’s Degree Completion
In the graphs above and the ones that follow, the two lines plot the degree outcomes for students who received a STBF award and those who did not, including a 95 percent confidence interval. The vertical axis shows the share of applicants who completed the indicated type of degree (bachelor’s or associate) by each year after application.
The underlying student sample varies by year; each year includes only the students who have been in school long enough to have completed the indicated years of schooling. For instance, the year-six spring results above include only students who were in the 2012 and 2013 cohorts and thus eligible for a scholarship six or more years ago at the time of publication.
Four-year students with and without STBF awards complete bachelor’s degrees at the same rate until year five. By year six, STBF awards increase bachelor’s degree completion rates by 8.4 percentage points.
The impact of STBF awards on students who indicated a two-year school as their first choice is less straightforward. It appears that STBF awards decrease associate degree completion rates by approximately 7 percentage points. This is likely caused by students who receive a STBF award moving from two-year schools to four-year schools. Although students initially indicate interest in a two-year program, an award from STBF may ease financial constraints and appears to prompt some students to complete a bachelor’s degree instead.
Bachelor’s Completion by Campus Strata
University of Nebraska at Omaha award recipients are more likely to complete their bachelor’s degree than their peers who do not receive an award. 62 percent of students who received a scholarship completed the requirements for a bachelor’s degree, compared with 48 percent of students who did not receive an award, a 15 percentage-point difference.
Differences in completion rates at the other University of Nebraska campuses (Kearney and Lincoln) and State Colleges are much smaller, and the confidence intervals indicate that these differences are not statistically significant.
In the figures that follow, the research team looked at bachelor’s degree effects on four-year applicants broken into subpopulations based on ACT score, race, Pell grant eligibility, and first-generation status.
Scholarships Increase Bachelor’s Degree Completion Rates for Historically Disadvantaged Students
STBF awards have a particularly large and positive effect on bachelor’s degree completion rates for some student subgroups: those with below-median ACT scores,* nonwhite students, Pell grant-eligible students, and first-generation college students (whose parents don’t have a bachelor’s degree). The scholarship produces a 14, 12, 10, or 10 percentage point increase in degree completion rates, respectively.
STBF awards led to large gains in college enrollment, persistence, and degree completion. Students who receive a STBF award spend more time in school, are more likely to enroll in a four-year program than in a two-year program and are more likely to complete a bachelor’s degree within six years.
These gains are largest for low-income, Pell grant–eligible applicants, and for applicants with lower high school test scores. Our findings regarding high school performance on standardized tests highlight the paradox of merit aid: awards based on past achievement are less likely to generate large gains in college completion rates than awards made to applicants who appear less ready for college.
As the current cohorts age, we will continue to study the scholarship program’s effects across demographic groups and on postgraduate labor market outcomes. Evidence on the long-term effects of financial support on both individuals and groups should also grow more conclusive. We will update this site as we obtain new results.
More information can be found in the following papers and briefs:
- Joshua Angrist, David Autor, and Amanda Pallais. "Marginal Effects of Merit Aid for Low-Income Students." Discussion paper 2020.06 (Cambridge, MA: School Effectiveness and Inequality Initiative, 2020).
- SEII. "The Effect of Financial Aid on Low Income, College Ready Students." Cambridge, MA, 2020.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Evaluating Post-Secondary Aid: Enrollment, Persistence, and Projected Completion Effects." Working paper 23015 (Cambridge, MA: National Bureau of Economic Research, 2016).
- J-PAL. "Evaluating Post-Secondary Aid: Enrollment, Persistence, and Projected Completion Effects." Cambridge, MA, 2016.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Evaluating Econometric Evaluations of Post-Secondary Aid." American Economic Review: Papers & Proceedings 105, no. 5 (May 2015): 502–7.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Leveling Up: Early Results from a Randomized Evaluation of Post-Secondary Aid." Working paper 20800 (Cambridge, MA: National Bureau of Economic Research, 2014).
The School Effectiveness and Inequality Initiative (SEII) is a research program based in the Massachusetts Institute of Technology (MIT) Department of Economics.
This research was led by Joshua Angrist (Ford Professor of Economics at MIT and co-director of SEII), David Autor (Professor of Economics at MIT and co-director of SEII), and Amanda Pallais (Professor of Economics at Harvard University), with expert implementation assistance from the staff of the Susan Thompson Buffett Foundation. The provost’s office at the University of Nebraska, the Nebraska State College System, and Nebraska’s community colleges provided integral support for the evaluation.
A team of researchers based at the University of Southern California are conducting a mixed-methods study on the effects of the STBF scholarship’s Learning Communities support. Visit their website here.