The Rankings Trap

One evening in September 2023, while traveling through in Southeast Asia on an admissions recruitment trip for my university, U.S. News published its 2024 college rankings. My colleague and I stared at the new list in disbelief, eyeing what turned out to be the largest year-over-year rankings shift in recent history. 

California State University, Fresno jumped 65 places.
Tulane University dropped 29 places.
University of Texas at San Antonio jumped 92 places.
American University dropped 33 places.

The biggest winner? Southeastern Baptist Theological Seminary, having jumped 106 places in the U.S. News rankings in a single year.

What changed? It is probably fair to assume that the quality of education at Southeastern Baptist Theological Seminary didn't wildly improve in that year alone, nor did American University become demonstrably worse.

What did shift was the formula that U.S. News used to build their rankings. This was not inherently bad. But the extent of its impact on student decision-making that year is hard to overstate. At the end of the day, that's probably a disservice to students.

Here are a few of my thoughts on the minefield of college rankings:

They can be useful

Like it or not, rankings are some of the first pieces of information that students and parents tend to look up when they begin their college search process. This intuition isn't totally unfounded: rankings formulas are based on some useful data points related to institutional resources for students and post-graduation outcomes. If college A is ranked 100 places higher than college B, it's probably fair to assume, at least broadly speaking, that there are meaningful differences between the two institutions in things like resources, graduation rates, and the range and depth of opportunities available to their students.

But...

...that's not how we actually tend to use college rankings. Instead, I've watched countless students turn them into a kind of scoreboard: My school was ranked 23rd last year and yours was 21st. Now mine is 22nd and yours is 25th. Therefore, my school is better than yours.

I have two problems with that line of thinking:

"Better"

The subjective side of rankings

The other problem I have with rankings hair-splitting is that some of the data that goes into determining rankings is less concrete and objective than you might think. For context, here's what went into U.S. News' formula for its 2027 college rankings.

Looking at those factors, we see some reasonable-looking metrics. Graduation rate is weighted at 16%. First year retention rate is 5%. Same for "college grads earning more than a high school grad." Faculty salaries are 8%. I think it's fair to say that these factors are correlated with the quality of an institution. (Though let's admit that the specific weights of each are relatively arbitrary—it's not hard to imagine how changing retention rate to 10% and slightly reducing the weight of faculty salary and graduation rate could heavily impact the rankings.)

But the factor that holds the heaviest weight in the U.S. News rankings recipe is something called "peer assessment," which is responsible for 20% of their formula. Here's the description of this factor from their own website:

"This is a measure of how a school is regarded by top administrators at other institutions and accounts for qualitative attributes of schools that may not be fully captured by the other ranking factors. A school's peer assessment score is calculated from the weighted, rolling average of ratings on academic quality it received by presidents, provosts and deans of admissions – or officials in equivalent positions – at institutions submitting a peer assessment survey administered by U.S. News...Each survey respondent was asked to rate the overall academic quality of peer schools' undergraduate academic programs on a 5-point scale: outstanding (5), strong (4), good (3), adequate (2) or marginal (1). "

This might not sound particularly interesting, but here's what they're saying: the largest factor that influences their college rankings is a survey that they send out to college officials, asking them to simply rank the "quality" of other colleges. These officials may know little or nothing about many of the schools they're being asked to rank, but they pick a score on a one-to-five scale.

And that's it. One fifth of the entire methodology is, as the kids would say, "vibes."

So the difference between schools ranked 30th and 34th? Or 16th and 25th? Or 80th and 90th? Nothing...apart from them simply being different institutions with different programs and campuses and extracurriculars and locations and student bodies and professors...and everything else.

And that's why we talk about the importance of finding the right fit school for you.