|
League Tables as Policy Instruments
Jamil Salmi and Alenoush Saroyan
Jamil Salmi is coordinator of the World Bank Tertiary Education Thematic Group. Alenoush Saroyan is professor in the Department of Educational and Counseling Psychology at McGill University, Montreal, Canada. The findings, interpretations, and conclusions expressed in this article are entirely those of the authors and should not be attributed in any manner to the World Bank. E-mail: jsalmi@worldbank.org.
In 1963, at the University of California at Berkeley, faculty and the administration objected when the radical student newspaper on campus, Cal Reporter, took the initiative to publish student evaluations of courses and professors. Despite this initial resistance, student evaluations have become part of many universities' internal accountability mechanisms. Demands for increased accountability come not only from students but also from other stakeholders—such as governments, employers, and the public at large. Institutional rankings by league tables constitute one way of meeting the keen interest in information about the quality of universities.
League tables use a wide range of indicators to measure the system's setup (input variables), function and efficiency (process variables), and productivity and impact (output variables)—relative to the performance of other universities and programs.
The expansion of league tables and ranking exercises has not gone unnoticed by the various stakeholders, and the reaction they elicit is rarely benign. In some countries (Argentina, Nigeria, and India), the ranking exercise is undertaken as part of the accreditation process. With leagues tables becoming something of a growth industry, their accuracy, relevance, and usefulness have become issues of concern. Do they have any beneficial use for public policy, accountability, and consumer information? This article examines ways in which league tables have been used internationally to enhance quality.
Germany
In 1990, after the fall of the Berlin wall, teams of academics from the West German Science Council were given the task of evaluating their counterparts at East German universities. It was realized that in the absence of a tradition of evaluation in West-German universities the mission would call for inventing an appropriate methodology. More recently, the ranking exercise conducted since 1998 by the Center for Higher Education Development (Centrum für Hochschulentwicklung—CHE), an independent policy research agency, has become the first comprehensive system providing a panorama of quality indicators in Germany. Instead of calculating a global ranking of institutions based on weighted indicators as do the Times Higher Education Supplement and Shanghai Jiao Tong University (SJTU), CHE presents detailed survey data from thousands of teachers and students as well as third-party data dealing with the universities and technological institutes separately. One can look at the standing of each university or even a specific academic subject, based on a particular indicator or set of indicators. Readers can even create their own ranking based on the indicators most relevant to them. The approach developed by CHE presents the additional advantage of avoiding data biases linked to self-reporting by universities. Austrian and Swiss universities have recently joined this exercise, accepting benchmarking against German universities.
Pakistan
The ranking of universities in Pakistan developed out of a direct mandate given to the Higher Education Commission (HEC) in 2002 to evaluate the universities in a way that promoted rapid and comprehensive development of the entire tertiary education system, particularly to support the country's place in the world economy. By comparing the inputs and outputs of the nation's institutions, Pakistan has established a mechanism for rewarding excellence and investing in the strengthening of institutions that need improvement. The five main ranking criteria used by HEC (faculty qualifications, research output, students, facilities, and finances) are similar to ranking indicators in many other nations.
The advisory committee overseeing this ranking exercise, comprised of HEC officials and university representatives, had to decide whether the results should be made public or not. Under vehement protests from one of the leading vice-chancellors, whose (public) university had scored very low, the committee agreed not to publish the results. What HEC did instead was to share key benchmarking data with each university, notably its relative position against each criterion used in the rankings. For example, university X was told that, with respect to the proportion of professors with a doctorate, it scored in the lowest quartile compared to all universities in Pakistan. Despite the general outcry against the publication of the rankings' results, this experience has forced the universities to take data collection much more seriously.
The Germany and Pakistan examples illustrate that, in countries without established evaluation or accreditation mechanisms, rankings can be used effectively to monitor and enhance quality. Finally, it is interesting to note that rankings are not used only by governments in their national context but also increasingly in an international perspective. In Mongolia and Qatar, for instance, the authorities have decided to restrict scholarships for studies abroad to students admitted in a highly ranked university.
Public Initiatives
The press is often criticized for using rankings as a gimmick to boost sales. However, the mass media can play a genuine educational role by making relevant information available to the public, especially in countries lacking any form of quality assurance mechanism. In Poland, when the transition to the market economy started in the early 1990s and many private education institutions began to operate, there was a thirst for information about the quality of these institutions, which pushed the owner of the Perspektyvy magazine to initiate the country's first ranking. Similarly, in Japan for many years the annual ranking published by the Asahi Shimbun newspaper fulfilled an essential quality assurance function, in the absence of any evaluation or accreditation agency.
Colombia
Colombia was the first country in Latin America to set up a national accreditation system in the mid-1990s, but the number of programs reviewed by the new accreditation agency remained relatively low in the first years because the accreditation law made the process voluntary and the most prestigious universities, public and private, did not feel any compulsion to participate. After the country's main newspaper, El Tiempo, published in 2000 the full list of accredited programs to help students choose among the various offerings, the universities have felt increasing pressure to join the accreditation process.
France
Another important merit of rankings is to stimulate public discussions around critical issues affecting the tertiary education system. A good example is the debate that started in France when the SJTU world rankings were published for the first time. After observing that the best French university was ranked 65th, the daily paper Le Monde ran an article, "The Great Misery of French Universities," that started a widespread discussion. One of the principal problems identified was the fact that French universities are not allowed to select the most academically qualified students, unlike the practice in more successful university systems in the United States, the United Kingdom, or Japan. The other important factor is the absolute lack of competition among universities.
Brazil
In 1996 the Ministry of Education in Brazil introduced an assessment test intended to compare the quality of undergraduate programs across universities. It could be described as a ranking exercise categorizing university programs based on the average score of their participating students. While the test (Provão) at first met with a lot of resistance, over time it became more accepted and increasingly influenced students in their choice of tertiary institution.
Nigeria
Similarly, in Nigeria, after the National Universities Council initiated a ranking of professional programs in 2001, even going as far as closing down a number of programs, private-sector employers started to regain confidence in local universities and to hire graduates of the highest-ranked programs.
The Way Forward
Governments and the public at large are increasingly preoccupied with the relative performance of tertiary education institutions. Some countries are striving to have "world-class universities" that will spearhead the development of a knowledge-based economy. Others, faced with a shrinking student population, struggle to attract more fee-paying foreign students. Just as scarcity, prestige, and excellence dominate the purchase of consumer goods, students are also looking for indicators to identify the best universities. Thus, regardless of their controversial nature and methodological shortcomings, university rankings are unlikely to disappear.
The following general recommendations, developed through our analysis of the recent international experience with league tables, can help make the ranking exercise beneficial to institutions, governments, students, parents and the public at large.
Clarify what the ranking measures. Despite the ambiguities surrounding the construct of quality, the rankers should specify what they do or do not measure, the goals, and the audiences for whom they do the ranking. The validity, reliability, and comprehensiveness of selected indicators can be discerned in light of these specifications and by examining the scope of the academic tasks (e.g., teaching, research, etc.) and the types of institutions being assessed. Furthermore, the rankers should make the raw data on which they base the ranking widely available and the calculation process transparent so that their rankings can be verified independently.
Use multiple indicators and measures rather than a single, weighted ranking. League tables should use a wide range of indicators, placing greater emphasis on output and outcome indicators to ensure that every dimension of quality gets factored in the evaluation. Multiple sets of indicators will yield multiple scores rather than a global score, thus bringing to light areas of strengths as well as areas of weaknesses.
Compare similar programs or institutions. Because of their methodological limitations, rankings are more meaningful when the unit of comparison is smaller. Ranking programs is thus preferable than ranking institutions.
At the institutional level, use rankings for strategic planning and quality improvement purposes. Tertiary education institutions that look at detailed ranking data for benchmarking purposes can use specific indicators to understand the determinants of their performance and then work toward improving the quality of teaching and research.
At the government level, use rankings to stimulate a culture of quality. In countries that do not yet have a well-established quality assurance system, rankings can be used as a proxy for quality. Similarly, at the international level, in the absence of a single global quality assurance agency, ranking systems take on some characteristics of a quality regulator for international students.
Use rankings as one of the instruments available to inform stakeholders. Rather than being considered as the ultimate measure of quality and/or relevance, rankings should be complemented by information on accreditation and labor market outcome data collected through surveys of employers and tracer surveys. The results of league tables can also serve to generate a national debate about long-term strategic priorities and policies for tertiary education.
[Online] Available:
http://www.bc.edu/bc_org/avp/soe/cihe/newsletter/Number47/p2_Salmi_Saroyan.htm
|
