Emory Report

April 20, 1998

 Volume 50, No. 29

First Person

Chace points out problems
with U.S. News rankings

In recent weeks, the University has once again complied with the annual request for information from the editors of U.S. News & World Report. The magazine will use these data, as well as information they gather from other sources, to construct the fall "ranking" they title "Best National Universities." I want to report to you that U.S. News is once again altering its methodology; one such alteration was the subject of my recent letter to James Fallows, editor of U.S. News.

Although the ranking seems to track relative change from year to year, it is in fact based each year on different variables weighted in different ways. I wrote to Fallows to highlight one change U.S. News has announced for the coming fall. My March 10 letter is reprinted here, following a few examples of some earlier changes and their possible effects.

One earlier change involved a variable U.S. News calls faculty resources. Before 1995 faculty resources counted for 25 percent of the total value of the U.S. News ranking "equation." In 1995 the value of faculty resources was reduced to 20 percent of the total value. Thus schools with strong support for faculty appeared to have declined, while schools with a weaker showing on this variable appeared to have held their own or grown stronger. In all likelihood, little had changed at any of the universities on the list.

Another earlier change concerned a variable U.S. News calls value added. To calculate value added, the magazine uses the average SAT score and its calculation of expenditure per student on instruction to construct each school's "projected graduation rate." This "projected graduation rate" is compared with the actual graduation and retention rates, and the difference becomes value added. Before 1997 value added did not exist. In 1997 U.S. News created it and altered the weights of other variables to give it value. This construction has, I believe, a double-dipping effect. Those variables that determine value added count twice-once on their own and once as part of value added.

Which brings us to the case in point for this year. As detailed in my letter to Mr. Fallows, this year the variable graduation rate is likely to lead to confusion. Although U.S. News has gathered some form of this information in the past, this year the magazine has changed its method for collecting this information. Once again, the ranking will reflect the way in which definitions have been altered but not the way in which reality has changed. Let us reflect on the fact that universities are slow to change, and let us acknowledge that this is the one reality on which we can rely.

Dear Mr. Fallows:

I am writing to share with you my views of your "Best Colleges" survey and of those items that deal with, as you term it, "graduation rate." As I understand it, these items, to which we must respond, conform to a federal questionnaire we in universities must complete. New this year, the questionnaire is designed to gather information on the number of students who pass through an institution during the course of their undergraduate careers. I believe the form has its origins in requirements made of Division I schools of the NCAA. (I note that Emory is not among these, as our students are admitted only on the basis of academic ability and achievement, and participate in our varsity sports programs as an extracurricular endeavor. Thus Emory is classified as a Division III school.)

We can and will supply to you the numbers of students who enter and then graduate, die, become disabled or enter the armed forces. However, there is a very important group about which we cannot inform you with the rigor you require. These are students who transfer and then graduate from other institutions in the six years specified by you to be counted in the graduation rate.

To count these graduates, we would have to generate, as you require, written documentation of each student's graduation from another institution. Assembling such documentation would require at least the full-time attention of one professional-one person entirely devoted to a task wholly unrelated to advancing either the mission of Emory or the well-being of our students. When I weigh the cost against such investments as adding a young scholar in the classroom, a skilled researcher to the information resources staff or perhaps two advisers to academic services, it is an alternative I cannot consider. Therefore, Emory will not conform to your requirement. As a consequence, we are now prepared to suffer in your fall ranking, even while believing that the real quality of an Emory education will not have changed one iota. As a further consequence, we will have some explaining to do to people who pay attention to your survey. We will have to say, as we now often say, that the many changes over time in the survey results are not a consequence of changes in the institutions themselves, but continual alterations in the formula you employ.

You should know, however, that we do track the progress of our students, but in more informative ways than you require. The reasons most students leave Emory are very good ones. Some come seeking a liberal arts foundation for an eventual degree in architecture or engineering, for example, and these are programs we do not offer.

In short, I believe your question is off the mark (and am further dismayed by the fact that the response counts twice in your formula). I again hope, as I hoped in an earlier letter I sent to you, that you will review the ways in which the questionnaire, and the formula you use and revise, do not truly probe genuine issues of educational excellence, but instead create a "game" that universities are compelled to play-some honestly and some not so honestly-as they see their reputations bounce up and down.

With some serious and widespread misgivings, we will continue to play the game. We will do so because many people who are important to us pay attention to the survey and do so despite our misgivings. But we will play it honestly, now preparing to suffer the consequences for so doing.

Thank you for your attention to this letter. Best wishes as you enter your second year at U.S. News.

William M. Chace

Return to April 20, 1998 Contents Page