Evaluating College Support
A study of the effects of targeted college aid offered by the Susan Thompson Buffett Foundation.
The Susan Thompson Buffett Foundation (STBF) offers scholarships to Nebraska high school graduates who attend the state’s public colleges and universities.
STBF is one of the largest private funders of college scholarships in the country. Awards cover the full cost of tuition and fees (plus $500 toward books) to attend college at any Nebraska state university or college. The scholarships can provide more than $70,000 toward five years of study.
Many of the scholarship winners who attend one of the three University of Nebraska campuses (in Lincoln, Kearney, or Omaha) also participate in Learning Communities, which provide academic and personal support in addition to social programming. The research team is tracking the progress of NU students who do and do not participate in Learning Communities and will share results that isolate the effects of Learning Communities in future research.
STBF provides over $40 million in aid every year to more than 4,000 students.
This study includes applicants to the STBF scholarship program between 2012 and 2016. In 2012, more than 2,000 high school students in Nebraska applied for a STBF scholarship.
In response to outreach and advertising, applications grew to almost 4,000 annually by 2016. From 2012 through 2016, roughly 16,500 students applied for a STBF scholarship.
To be eligible for the scholarship, applicants must have had an unweighted high school grade point average of 2.5 or higher and documented financial need (an expected family contribution below $15,000 in 2012 and $10,000 in 2013 through 2016).
Of the 16,500 applicants, 11,000 were eligible for a STBF scholarship. We grouped these eligible applicants by their first-choice campus. The two-year group (about 15% of eligible applicants) includes applicants who said that, if admitted, they would attend a two-year college, while the four-year group (85% of eligible applicants) includes applicants who targeted a four-year college.
Applicants were scored based on academic achievement, financial need, a personal essay and letters of recommendation.
The highest scoring applicants were awarded aid. The lowest scoring applicants did not receive an award.
Our study focuses on the middle group, applicants who were all equally deserving of an award. Between 2012 and 2016, nearly three-quarters of eligible applicants were in this middle group.
STBF is interested in understanding the impact of the scholarship on student outcomes such as college enrollment, persistence, degree completion, and college debt.
How should we evaluate a scholarship program? One might simply compare applicants who receive scholarships with those who don't, but this isn’t the best approach. STBF aid is awarded partly on merit, like many other programs, such as state college aid for students with a strong high school record. High-performing students are likely to be awarded a scholarship, while low-performing students are not. The students who receive scholarships would likely complete college at a higher rate even without aid.
To solve this problem, the research team deployed a partially randomized design that focuses on applicants in the middle of the pool. As the middle group of applicants were deemed all equally qualified for an award, STBF randomly awarded scholarships to up to half of this group in each year. Randomization is fair in this context because the number of applicants with scores in the middle range exceeds the number of awards available.
Random assignment ensures that, on average, applicants in the middle group who did and did not receive aid were similar before STBF awarded scholarships.
Statistical comparisons show that the applicants in the middle group who were and were not offered STBF awards are indeed similar. In particular, they have similar:
- Demographic characteristics, such as gender and race;
- High school grade point averages and composite ACT scores;
- Family incomes and parental education backgrounds.
Since these two groups were nearly identical before the award and the awards were assigned randomly, any difference in outcomes reflects the causal effect of the STBF scholarship.
These results are part of an ongoing research study. Our current analysis looks at enrollment and college completion through spring 2019. Later analyses will include more data as subsequent cohorts of students in the study reach the date of expected college completion. The results that follow separate the sample into randomized applicants who targeted four-year schools and randomized applicants who targeted two-year schools.
* Small discrepancies between the point estimates referenced in the text and the graph values are due to rounding. The downloadable data contains the precise estimates.
The gray numbers above each pair of bars indicate the percent change in each aid category for applicants who receive an award. For example, the graph shows a 32 percent increase in total aid for award winners. The small interval markers overlapping the top of each bar indicate a 95 percent confidence interval. This means that if the STBF scholarship study were repeated 100 times with different random samples of students, each time calculating the 95 percent confidence interval for the new sample, we would expect that 95 percent of those confidence intervals would contain the true population mean.
Susan Thompson Buffett Foundation (STBF) award amounts are calibrated to match the cost of tuition and fees for each student based on their part-time or full-time status, plus $500 for books. Each pair of bars shows the effect of a STBF scholarship on the category listed at the bottom of the graph in the applicant’s freshman year. The two bars to the far left indicate that STBF award winners on average receive $8,158 in aid compared to their peers who receive $0 in aid from the Foundation. Compared with students who did not receive an award, award recipients receive double the grant aid and thus rely less on federal loans and work-study programs.
46 percent of award recipients finished their fourth year without any federal loans. By comparison, only 29 percent of those who did not receive an award had no federal loans after four years. STBF awards increased the share of students who had no federal loans in year four by 17 percentage points.
In the graph above and the ones that follow, the shaded area indicates a 95 percent confidence interval. This means that if the STBF scholarship study were repeated 100 times with different random samples of students, each time calculating the 95 percent confidence interval for the new sample, we would expect that 95 percent of those confidence intervals would contain the true population mean. In this chart, the shaded area around the line for students who received a scholarship does not overlap with the line for students who were not awarded a scholarship, suggesting that our results are not chance findings.
STBF awards boosted enrollment and cut dropout rates in half after four years. Enrollment among students who did not receive a scholarship fell from 96 percent in students’ first semester to 78 percent in the fall semester of students’ fourth year, a decline of 18 percentage points. Enrollment among students who were awarded scholarships fell by only 13 percentage points over the same period.
STBF awards also increased enrollment for students who targeted a two-year college. More than half of STBF award winners were still in school in the fall, four years after winning the award, compared with less than half of students who did not win an award. Some students who start in a two-year program transfer to a four-year school with continued STBF support.
In the graphs above, each dot represents a cluster of students with similar ACT scores. On the horizontal axis, ACT scores (which range from 1 to 36) are converted to percentile values. Each cluster represents 10 percent of the control or treatment group.
These graphs show the relationship between students’ ACT scores and college enrollment. The first graph shows the share of students enrolled in year one at any postsecondary institution (i.e., a two- or four-year college). Enrollment rates among award recipients and other students are not very different: regardless of ACT score, both groups enroll in college at roughly the same rate.
Initial enrollment in four-year colleges differs by ACT score. Award recipients with lower ACT scores are more likely to enroll in a four-year college than are similar-scoring students who did not receive an award. Conversely, award recipients with high ACT scores enroll in four-year colleges at the same rate regardless of whether they receive an award. STBF awards appear to have shifted students with below-median ACT scores from two-year colleges to four-year colleges.
In the graphs above and the ones that follow, the two lines plot the degree outcomes for students who received a STBF award and those who did not, including a 95 percent confidence interval. The vertical axis shows the share of applicants who completed the indicated type of degree (bachelor’s or associate) by each year after application.
The underlying student sample varies by year; each year includes only the students who have been in school long enough to have completed the indicated years of schooling. For instance, the year-six spring results above include only students who were in the 2012 and 2013 cohorts and thus eligible for a scholarship six or more years ago at the time of publication.
Four-year students with and without STBF awards complete bachelor’s degrees at the same rate until year five. By year six, STBF awards increase bachelor’s degree completion rates by 8.4 percentage points.
The impact of STBF awards on students who indicated a two-year school as their first choice is less straightforward. It appears that STBF awards decrease associate degree completion rates by approximately 7 percentage points. This is likely caused by students who receive a STBF award moving from two-year schools to four-year schools. Although students initially indicate interest in a two-year program, an award from STBF may ease financial constraints and appears to prompt some students to complete a bachelor’s degree instead.
University of Nebraska at Omaha award recipients are more likely to complete their bachelor’s degree than their peers who do not receive an award. 62 percent of students who received a scholarship completed the requirements for a bachelor’s degree, compared with 48 percent of students who did not receive an award, a 15 percentage-point difference.
Differences in completion rates at the other University of Nebraska campuses (Kearney and Lincoln) and State Colleges are much smaller, and the confidence intervals indicate that these differences are not statistically significant.
In the figures that follow, the research team looked at bachelor’s degree effects on four-year applicants broken into subpopulations based on ACT score, race, Pell grant eligibility, and first-generation status.
STBF awards have a particularly large and positive effect on bachelor’s degree completion rates for some student subgroups: those with below-median ACT scores,* nonwhite students, Pell grant-eligible students, and first-generation college students (whose parents don’t have a bachelor’s degree). The scholarship produces a 14, 12, 10, or 10 percentage point increase in degree completion rates, respectively.
*The Nebraska Median ACT score is 21.
Do the benefits of an STBF award outweigh the costs?
We offer two views of scholarship program costs.
The first is the direct cost to the Susan Thompson Buffett Foundation, which we call funder cost. The average student targeting a four-year public university costs the foundation $32,250 over the course of the student’s post-secondary schooling. For state colleges, which are not part of the University of Nebraska system, that cost is $26,770.
The second measures the incremental cost of attendance, that is, the average cost of schooling for students receiving an STBF scholarship (the treatment group) minus the average cost of educating students receiving no STBF scholarship (the control group). The treatment cost is higher than the control cost when STBF students spend more time enrolled in school and/or choose to attend a more expensive degree program as a consequence of being awarded the scholarship. The graph below breaks down incremental costs by institution.
The pink sections atop the blue bars indicate the incremental cost of attendance for students targeting each school. The incremental cost of attendance is the average cost of schooling for students receiving an STBF scholarship (the teal bars) minus the average cost of schooling for students receiving no STBF scholarship (the blue bars). The red bars represent the funder cost for each school.
The graph above shows the funder cost and cost of attendance for those who received an STBF scholarship and those who did not. It depicts the variation in the incremental cost of attendance among four different schools. The highest incremental cost is for students targeting the University of Nebraska-Omaha. For STBF students, the average cost of attendance is $30,870, compared with $26,030 for students who are not on STBF scholarship. Thus, the incremental cost of attendance is $4,840.
In part, the extra cost stems from STBF recipients spending more time enrolled in school, compared with control students. Also affecting incremental cost are students who end up at a more expensive school than the one they had initially targeted. For instance, upon receiving the scholarship, some students who targeted University of Nebraska-Omaha may switch to University of Nebraska-Lincoln, which has higher tuition and fees than its Omaha counterpart. The lowest incremental cost of attendance among schools included is for students targeting state colleges (as distinct from universities), yielding an incremental cost of attendance of -$290, meaning that STBF scholarship recipients spend slightly less on average than the non-recipients.
We assess the benefits of the STBF scholarship based on predicted lifetime earnings, using data from the American Community Survey for Nebraska-born respondents, ages 18 to 65. We calculate wages at each age based on degree attainment and potential work experience. Students who earn a bachelor’s degree will, on average, earn more over their lifetime than those with an associate degree. We calculate potential work experience under the assumption that the average work life spans ages 18 to 65. A student who takes five years to complete a bachelor’s degree would have one less year of potential work experience than one who takes four years.
With earnings estimates, we can compare costs and predicted benefits for those receiving a scholarship and those who did not among different groups of students.
In descending order by grouping, the graph shows the benefits – in terms of additional lifetime earnings (represented by dark green circles) – projected to accrue to STBF students compared with their peers who were not awarded a scholarship. Funder cost is represented by red circles; the incremental cost of attendance by pink circles. For example, among all female students, the average funder cost is $32,578, the gain in average predicted earnings is $18,304, and the average incremental cost of attendance is $2,043. When scrolling over each student group, a corresponding group is highlighted. For example, scrolling over the Omaha resident group simultaneously highlights the not Omaha resident group.
The sample used for this figure includes the 2012-2014 scholarship cohorts who selected a four-year college as their top choice in their scholarship application. The “two-year college alternative” group refers to students who listed a community college as an alternative (second- or third-choice) target school. Pell-eligible means that the students were eligible to receive the Federal Pell Grant due to exceptional financial need.
The graph above shows different cost and benefit measures, allowing us to compare the effect of the STBF scholarship among different groups. The average predicted lifetime earnings gain is the difference between predicted lifetime earnings of those who received an STBF scholarship and those who did not. For all groups, the benefits of the program (shown in green) outweigh the incremental cost of attendance (shown in pink). This means that scholarship recipients are predicted to earn more over their lifetime than the extra cost of attendance that was incurred because of receiving the scholarship.
Importantly, the benefits of the scholarship program outweigh not only the incremental cost of attendance but also the funder cost for some of the student groups that have been historically underrepresented in higher education, as seen at the top of the graph. These groups include Omaha residents, students who target University of Nebraska - Omaha (41.7% of their total undergraduate population was a first-generation college student in the 2020-2021 academic year), those with below median ACT scores, those who listed a community college as an alternative target school, and those with a below median GPA. These groups also have some of the highest award-induced bachelor’s degree completion rates, as discussed earlier.
Benefiting most are Omaha recipients. They are predicted to earn an average of $39,151 more during their working lives than their peers who were not awarded a scholarship. That group on average provides the biggest return on STBF investment: $7,757 above the $31,394 foundation cost of educating the average Omaha recipient.
Among STBF recipients in groups that are predicted to realize the lowest gains are those with an above median high school GPA, those targeting state colleges, those who are not Omaha residents, and those with at least one parent who attained a BA. The STBF scholarship does not have a high payoff for these students because they are likely to complete the bachelor’s degree even without STBF support. Even for these groups, however, predicted earnings gains exceed the incremental cost of attendance, meaning that the additional college investment is likely to pay off during adulthood. Coupled with the findings on the subgroups with high predicted earnings gains, these results strongly suggest that increased targeting of financial aid awards is likely to enhance aid impact.
We emphasize that because we have only predicted future earnings at this time, rather than actual earnings outcomes for each student, these results are statistical ‘best guesses’ at expected earnings. The realized causal effects of STBF scholarship awards on adult earnings will not be known for at least a decade.
STBF awards led to large gains in college enrollment, persistence, and degree completion. Students who receive a STBF award spend more time in school, are more likely to enroll in a four-year program than in a two-year program and are more likely to complete a bachelor’s degree within six years.
These gains are largest for low-income, Pell grant–eligible applicants, and for applicants with lower high school test scores. Our findings regarding high school performance on standardized tests highlight the paradox of merit aid: awards based on past achievement are less likely to generate large gains in college completion rates than awards made to applicants who appear less ready for college.
As the current cohorts age, we will continue to study the scholarship program’s effects across demographic groups and on postgraduate labor market outcomes. Evidence on the long-term effects of financial support on both individuals and groups should also grow more conclusive. We will update this site as we obtain new results.
More information can be found in the following papers and briefs:
- Joshua Angrist, David Autor, and Amanda Pallais. "Marginal Effects of Merit Aid for Low-Income Students." The Quarterly Journal of Economics, Volume 137, Issue 2 (May 2022): 1039–1090.
- Joshua Angrist, David Autor, and Amanda Pallais. "Marginal Effects of Merit Aid for Low-Income Students." Discussion paper 2020.06 (Cambridge, MA: Blueprint Labs, 2020).
- Blueprint Labs. "The Effect of Financial Aid on Low Income, College Ready Students." Cambridge, MA, 2020.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Evaluating Post-Secondary Aid: Enrollment, Persistence, and Projected Completion Effects." Working paper 23015 (Cambridge, MA: National Bureau of Economic Research, 2016).
- J-PAL. "Evaluating Post-Secondary Aid: Enrollment, Persistence, and Projected Completion Effects." Cambridge, MA, 2016.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Evaluating Econometric Evaluations of Post-Secondary Aid." American Economic Review: Papers & Proceedings 105, no. 5 (May 2015): 502–7.
- Joshua Angrist, David Autor, Sally Hudson, and Amanda Pallais. "Leveling Up: Early Results from a Randomized Evaluation of Post-Secondary Aid." Working paper 20800 (Cambridge, MA: National Bureau of Economic Research, 2014).
Blueprint Labs is a research program based in the Massachusetts Institute of Technology (MIT) Department of Economics.
This research was led by Joshua Angrist (Ford Professor of Economics at MIT and co-director of Blueprint Labs), David Autor (Professor of Economics at MIT and co-director of Blueprint Labs), and Amanda Pallais (Professor of Economics at Harvard University), with expert implementation assistance from the staff of the Susan Thompson Buffett Foundation. The provost’s office at the University of Nebraska, the Nebraska State College System, and Nebraska’s community colleges provided integral support for the evaluation.
A team of researchers based at the University of Southern California are conducting a mixed-methods study on the effects of the STBF scholarship’s Learning Communities support. Visit their website here.