First Person
Chace points out problems
with U.S. News rankings
In recent weeks, the University has once again complied with the annual
request for information from the editors of U.S. News & World Report.
The magazine will use these data, as well as information they gather from
other sources, to construct the fall "ranking" they title "Best
National Universities." I want to report to you that U.S. News
is once again altering its methodology; one such alteration was the subject
of my recent letter to James Fallows, editor of U.S. News.
Although the ranking seems to track relative change from year to year,
it is in fact based each year on different variables weighted in different
ways. I wrote to Fallows to highlight one change U.S. News has announced
for the coming fall. My March 10 letter is reprinted here, following a few
examples of some earlier changes and their possible effects.
One earlier change involved a variable U.S. News calls faculty
resources. Before 1995 faculty resources counted for 25 percent of the total
value of the U.S. News ranking "equation." In 1995 the
value of faculty resources was reduced to 20 percent of the total value.
Thus schools with strong support for faculty appeared to have declined,
while schools with a weaker showing on this variable appeared to have held
their own or grown stronger. In all likelihood, little had changed at any
of the universities on the list.
Another earlier change concerned a variable U.S. News calls value
added. To calculate value added, the magazine uses the average SAT score
and its calculation of expenditure per student on instruction to construct
each school's "projected graduation rate." This "projected
graduation rate" is compared with the actual graduation and retention
rates, and the difference becomes value added. Before 1997 value added did
not exist. In 1997 U.S. News created it and altered the weights of
other variables to give it value. This construction has, I believe, a double-dipping
effect. Those variables that determine value added count twice-once on their
own and once as part of value added.
Which brings us to the case in point for this year. As detailed in my
letter to Mr. Fallows, this year the variable graduation rate is likely
to lead to confusion. Although U.S. News has gathered some form of this
information in the past, this year the magazine has changed its method for
collecting this information. Once again, the ranking will reflect the way
in which definitions have been altered but not the way in which reality
has changed. Let us reflect on the fact that universities are slow to change,
and let us acknowledge that this is the one reality on which we can rely.
Dear Mr. Fallows:
I am writing to share with you my views of your "Best Colleges"
survey and of those items that deal with, as you term it, "graduation
rate." As I understand it, these items, to which we must respond, conform
to a federal questionnaire we in universities must complete. New this year,
the questionnaire is designed to gather information on the number of students
who pass through an institution during the course of their undergraduate
careers. I believe the form has its origins in requirements made of Division
I schools of the NCAA. (I note that Emory is not among these, as our students
are admitted only on the basis of academic ability and achievement, and
participate in our varsity sports programs as an extracurricular endeavor.
Thus Emory is classified as a Division III school.)
We can and will supply to you the numbers of students who enter and
then graduate, die, become disabled or enter the armed forces. However,
there is a very important group about which we cannot inform you with the
rigor you require. These are students who transfer and then graduate from
other institutions in the six years specified by you to be counted in the
graduation rate.
To count these graduates, we would have to generate, as you require,
written documentation of each student's graduation from another institution.
Assembling such documentation would require at least the full-time attention
of one professional-one person entirely devoted to a task wholly unrelated
to advancing either the mission of Emory or the well-being of our students.
When I weigh the cost against such investments as adding a young scholar
in the classroom, a skilled researcher to the information resources staff
or perhaps two advisers to academic services, it is an alternative I cannot
consider. Therefore, Emory will not conform to your requirement. As a consequence,
we are now prepared to suffer in your fall ranking, even while believing
that the real quality of an Emory education will not have changed one iota.
As a further consequence, we will have some explaining to do to people who
pay attention to your survey. We will have to say, as we now often say,
that the many changes over time in the survey results are not a consequence
of changes in the institutions themselves, but continual alterations in
the formula you employ.
You should know, however, that we do track the progress of our students,
but in more informative ways than you require. The reasons most students
leave Emory are very good ones. Some come seeking a liberal arts foundation
for an eventual degree in architecture or engineering, for example, and
these are programs we do not offer.
In short, I believe your question is off the mark (and am further
dismayed by the fact that the response counts twice in your formula). I
again hope, as I hoped in an earlier letter I sent to you, that you will
review the ways in which the questionnaire, and the formula you use and
revise, do not truly probe genuine issues of educational excellence, but
instead create a "game" that universities are compelled to play-some
honestly and some not so honestly-as they see their reputations bounce up
and down.
With some serious and widespread misgivings, we will continue to play
the game. We will do so because many people who are important to us pay
attention to the survey and do so despite our misgivings. But we will play
it honestly, now preparing to suffer the consequences for so doing.
Thank you for your attention to this letter. Best wishes as you enter
your second year at U.S. News.
William M. Chace
Return
to April 20, 1998 Contents Page |