Table of Contents

INTERNATIONAL HIGHER EDUCATION

The Place of Citations in Today's Academy

Amanda Goodall
Amanda Goodall is a doctoral researcher at Warwick Business School, University of Warwick, UK, CV4 7AL. E-mail: amanda@amandagoodall.com.

Citations are references to authors in other academic papers as acknowledgment of their contribution to a specific research area. The field of bibliometrics was pioneered in 1955 by Eugene Garfield. He saw that information tools could be used by scholars to map scientific trends, assess the influence of individual papers, and of course trace the impact of their own work.

The Institute for Scientific Information (ISI) is the citation index most used today by the academic community. ISI includes bibliographic data on the sciences, social sciences, and more recently the arts and humanities.

The most common use of the database is for retrieving journal articles. However, increasingly bibliographic data are being used as a heuristic technique for assessing scientists’ work, and they are even being used by governments to evaluate whole nations’ research output. In today’s academy, citations have become a central indicator of quality and productivity, one that was unavailable a small number of years ago.

The use of such bibliographic data is important. It represents a shift in emphasis away from the quantity of journal articles, onto assessing the actual influence of academic work. One influential article matters more than 10 that nobody cites.

However, language biases have traditionally been shown to exist within ISI because of the dominance of English, although, as most international journals are increasingly producing English versions, language bias is steadily becoming less of a problem. There are those who assert that the global dominance of English is a form of cultural imperialism. But then, academic publications tend to follow research money, and the United States still has the largest research budget of any nation.

Weaknesses
Many deans, department heads, and recruitment committees are using citations as part of the assessment process involved in making new appointments. By going through databases like ISI, selection panels can check if a candidate’s work is making an impact by being cited by other researchers in their field.

To some, using citations as a measure of research quality is still a controversial practice. Critics assert that bibliographic data were not intended to be used as tools for measuring excellence but instead for information retrieval. Despite objections, bibliometric measures have become part of the evaluation process of academic research.

A common criticism of citations data is that an author can receive citations for bad work as well as for work that is viewed as good. This happens, some argue, because authors will often cite a piece of work that is receiving criticism and, therefore, the publishing author of “bad science” might accrue as many citations as someone publishing quality work. Of course, what is considered “bad science” may be hotly disputed, and these disputes themselves are part of the process of scientific development.

It is my belief that, in general, studies genuinely shown to be wrong or to have been falsified will not go on to generate high numbers of citations. Of course, there will always be exceptions. Papers that are repeatedly referenced negatively may generate a small bubble of citations, but it is unlikely that future work from the same author will receive a great deal of attention. Indeed, authors who write on bibliographic data on the whole agree that citations analyses are a reliable indicator of performance over long periods of time.

There are also problems with the accuracy of bibliographic data collection. Inconsistencies in methods of referencing and inaccuracies in citation statistics have been common factors. Self-citing presents another potential glitch, and this can take two forms: first, overciting one’s own work in academic papers and, second, self-citation in articles to try to raise a journal’s “impact factor.” Journal-impact factors measure the intellectual influence of journals by counting which ones are receiving the most citations (over the latest two years) by publishing authors. So, for example, if the most cited work is coming from articles in “Journal of Great Work” then this will increase the impact factor of that publication.

Traditionally, the status of an academic journal was built up through the reputation of editors, the editorial board, and the eminent scholars who published ground-breaking research between the covers of the periodical. Of course, time and again the most influential work failed to appear in the top publications. There have been many examples of articles that become the most cited yet that failed to be accepted by leading journals in their field.

When ISI first began to assess academic journals using this methodology, there were inconsistencies; and this is where the notion of self-citation returns. One study that looked specifically at this issue reported a significant correlation between self-citation levels and impact-factor scores across six journals in the field of anaesthesia. Some of these loopholes have been closed. Nevertheless, journal impact factors should still be applied with some caution.

Comparing Citations Across Disciplines
When using citations as any kind of measure of quality, it is important to recognize the huge differences between disciplines. For example, a very highly cited social scientist (say, one of Harvard’s best professors) might have a lifetime citation score of around 3,000–4,000, whereas a top molecular biologist could have a score of over 15,000–20,000. The discrepancies in citation levels across disciplines are demonstrated in the number of new cited references that appear in ISI every week. The sciences generate approximately 350,000 new cited references weekly, the social sciences 50,000, and the humanities 15,000.

Bibliometric indicators have been used more consistently across the sciences than in the humanities and social sciences. Such use is most evident in the natural and life sciences. These disciplines publish more journal articles and have a higher prevalence of coauthorship. In the social sciences, it is now quite common for there to be up to three authors attributed to an article, but any more is unusual—whereas in the sciences, coauthors can easily extend to the tens or twenties.

Another issue that skews disciplinary comparison is the publication rhythm and turnaround times of journals. Some medical disciplines have weekly journals; in history, the journals are often quarterly. In the discipline of economics it can take up to two years from the time an article has been accepted to the date of publication. In the arts and humanities, writing articles for journals is much less common. These disciplines tend more toward publishing monographs.

Conclusion
One of the most interesting, though possibly unsurprising, outcomes associated with the heightened awareness of citations is the extent to which they are being used to create league tables of top scientists. In 2005, Jorge Hirsch developed an h-index, essentially a method of counting citations, which he uses to identify and rank the most-cited physicists. For some scholars, counting one’s own citations has by repute become almost obsessional. I personally know of a physicist who checks his numbers every single morning.

It could be argued that this level of citations awareness is somewhat unhealthy and overly competitive. Then again, maybe it is inevitable in a world that celebrates those who are first—to identify a fact or explain a phenomenon. It is worth mentioning at this point that for those who would like to improve their citation levels there is a very quick and easy method. Ensure that you place all your academic papers (and others) onto your website with live links to the full text. This will not only generate a few more citations for you, but also, and more importantly, it will get your work out to other scholars and generally disseminate your ideas more widely. This must be good for science.

[Online] Available: http://www.bc.edu/bc_org/avp/soe/cihe/newsletter/Number43/p2_Goodall.htm