What's in a University Ranking?

The latest rankings rate UCSF ninth among universities worldwide - pretty impressive. Although the rankings - produced by Newsweek International - are among the first-ever international comparisons, there now are a rapidly growing number of groups across the globe that are developing university rankings - and the methods used often are criticized. What Are the Recent Ratings Based on?
"The Top 100 Global Universities" published last week by Newsweek International is based on criteria largely developed for two previous surveys by other groups. One set of earlier rankings was compiled by the Times of London. Their second annual "World University Rankings" were published in October 2005, part of the Times Higher Education Supplement. UCSF was ranked number 17 - 10th among all US universities on the list, and behind UC Berkeley among US public universities. The other rankings criteria adopted by Newsweek come from a lower-profile source, Shanghai Jiaotong University. Faculty members there, based on their own academic interests, started the rankings to find out how Chinese universities fared. In their "Academic Ranking of World Universities 2006," UCSF was rated number 18 - behind five other US public universities. How did combining criteria from two different rating processes - in which UCSF came out 17th and 18th - result in an elevation to 9th place in the largely derivative Newsweek International rankings? Rating Research Accomplishment
In all three sets of rankings, in each category the highest-ranked institution received a score of 100. Total scores for other institutions in each category are a percentage of the highest institution's tally. The ranking methodology used by the Chinese academics was focused entirely on measuring research strengths of universities across the globe. Harvard University ranked number one by every measure. Each of four different measures of faculty research accomplishment accounted for 20 percent of the rating:

• • Number of faculty receiving a Nobel Prize or the Fields Medal, the premier prize in mathematics (scores are proportionally reduced for prizes shared with researchers from other universities)
• • Number of papers published in Science and Nature
• • Number of researchers listed by Thomson Scientific (formerly ISI) among the most highly cited in their scientific academic fields over a period of 20 years
• • Articles in Science Citation Index Expanded, Arts and Humanities Citation Index, and double-counting of articles in the Social Science Citation Index

Two other categories received a weight of 10 percent in the Chinese rankings:

• • Number of alumni who won Nobel Prizes or Fields Medals
• • Size adjustment - total of all other tallies divided by the number of faculty

By these measures, what were the strengths and weaknesses of UCSF, which rated 16th among US universities and 18th overall? UCSF ranked seventh in publications in Nature and Science, and ninth in research accomplishment for its size. UCSF ranked 14th in the number of highly cited researchers. According to the Chinese academics' accounting, UCSF's two Nobel Prizes in medicine or physiology, unshared with other institutions, also ranked it 14th in the awards category. UCSF ranked 32nd in the category used to gauge strength of social science research - surprising for a small school best known for biomedical research and for training doctors, dentists, pharmacists and nurses. The biggest drag on UCSF's overall score in the Chinese rankings - even though it only accounted for 10 percent - was a score of zero for Nobel Prize-winning alumni. Because UCSF does not offer undergraduate degrees, it would be hard to compete in this category. The methodology used to compile the Times Higher Education Supplement list also included a statistical measure of research accomplishment. The gauge was the number of citations per faculty member, provided by Evidence Ltd, using data from Thomson Scientific for the years from 1995 to 2005. This criterion accounted for 20 percent of the Times rankings. By this measure, UCSF ranked sixth in the world. Peer Review and Employer Survey
The Newsweek International rankings assigned a weight of 16.67 percent to these criteria, borrowed from the Shanghai Jiaotong rankings:

• • Number of papers published in Science and Nature
• • Number of researchers listed by Thomson Scientific (formerly ISI) among the most highly cited in their scientific academic field over a period of 20 years
• • Articles in Science Citation Index Expanded, Arts and Humanities Citation Index, and double-counting of articles in the Social Science Citation Index

In addition, the Newsweek International rankings assigned a weight of 10 percent to each of these criteria, borrowed from the Times rankings:

• • Faculty/student ratio
• • Citations per faculty member (Thomson Scientific, 1995-2005 only)
• • Number of international faculty
• • Number of international students

It is likely that UCSF ranked higher in the Newsweek rankings than in either of the rankings from which it was largely derived because the categories where UCSF ranked highest - Nature and Science papers, faculty-student ratio, number of citations per faculty - were retained, while some categories in which UCSF ranked lower got tossed. UCSF, without undergraduates and with many research faculty, scored a whopping 91 in faculty-student ratio. But, as with most other US universities, scores for international faculty and international students were low. UCSF had about 4 percent as many international students and 6 percent as many international faculty as the London School of Economics. To round out their criteria, Newsweek included another measure not used in either of the two other rankings:

• • Number of volumes in the university library

UCSF has 864,000 volumes in its library, while Harvard has in the neighborhood of 15.5 million, according to Karen Butter, university librarian at UCSF. Expert Reviewers, Company Recruiters
Neither the Shanghai Jiaotong survey nor the Newsweek survey relied on surveys of experts in compiling their rankings. However, the Times Higher Education Supplement, to develop their 2005 World University Rankings, hired the consulting firm QS QuacquarelliSymonds Ltd. QS selected 2,375 "research-active" academics worldwide, who were asked to name top universities in their areas of expertise. UCSF did not appear to make a strong impression among these experts, scoring 24 in a category that carried a weight of 40 percent. In addition, QS queried companies they knew to be major recruiters of top graduates worldwide, with additional company names of top recruiters provided by universities. The companies were asked to provide names of 20 universities whose graduates they most prefer to hire. There were 333 responses. UCSF scored zero in this category, which carried a weight of 10 percent. These low scores among experts consulted for the Times rankings stand in stark contrast to the rankings that UCSF's training programs for doctoral scientists consistently have received from US News and World Report, perhaps the best known of the college rankers. US News ranks these graduate specialty programs based on nominations and ratings from department heads and directors of graduate studies at peer institutions. UCSF graduate training programs in biological sciences routinely place in the top 10. Ranking the quality of training programs for researchers and ranking the research itself are not the same thing, but the two are related. The US News survey results are more in line with UCSF research funding awarded by the federal National Institutes of Health (NIH). The NIH relies on panels of expert researchers - peers - to rank research proposals. For decades, UCSF has consistently ranked among the top five universities in NIH funding. Setting Standards
Clearly, what goes into a university ranking can vary greatly, and methodology problems are abundant even after the criteria are determined. Perhaps because they had no magazines to sell, the Chinese researchers were especially forthcoming about the methodological problems of their rankings: "Methodological problems include: the proportion of indicators on teaching and services, the weight of per capita performance, the type of institutions (comprehensive or specialized), the language bias in publications, the selection of awards and the experience of award winners. Technical problems include: the definition of institutions, the attribution of publications and awards, and the history of institutions." In 2004, the UNESCO-European Centre for Higher Education and the Washington-based Institute for Higher Education Policy (IHEP) convened a panel of individuals who develop or analyze rankings to foster debate and develop guidelines. The group issued a set of principles on ranking at the end of May. Jamie P. Merisotis, president of IHEP, said in a statement, "While colleges and universities cannot agree on whether or not rankings are a desirable practice, there is an increasing understanding that they are here to stay." Related Links: "UCSF's Global Status Attracts Foreign Students and Scholars"
UCSF Today, August 25, 2006 "UCSF Ranked Ninth Among Top Global Universities"
UCSF Today, August 23, 2006 "The Complete List: The Top 100 Global Universities"
Newsweek International Edition, August 2006 "Academic Ranking of World Universities - 2006"
Shanghai Jiao Tong University "World University Rankings 2005"
Times of London "Methodology and Quality Standards of University Rankings" Second Meeting of the International Rankings Expert Group (IREG), May 18-20, 2006, Berlin, Germany