Scholarly journal evaluation based on panel data analysis

Abstract

This paper proposes a new method for indicator selection in panel data analysis and tests the method with relevant data on agricultural journals provided by the Institute of Scientific & Technical Information of China. An evaluation exercise by the TOPSIS method is conducted as a comparison. The result shows that panel data analysis is an effective method for indicator selection in scholarly journal evaluation; journals of different disciplines should not be evaluated with the same criteria; it is beneficial to publish all the evaluation indicators; unavailability of a few indicators has a limited influence on evaluation results; simplifying indicators can reduce costs and increase efficiency as well as accuracy of journal evaluation.

Introduction

Research on journal evaluation methods is an important component of bibliometric studies. It tries to disclose regularities in the distributions of publications in disciplinary journals and quantitatively analyses developments and growth trends of scholarly journals so as to improve journal utilization. Journal evaluation exercises can, moreover, improve the quality of scholarly journals and promote their healthy development. In the 1960s Eugene Garfield, the founder and one-time president of the Institute for Scientific Information, conducted several large-scale statistical analyses of journal citations and reached the conclusion that the majority of citations were attributed to relatively few journals, while a minority of citations was spread over many journals. His work can be viewed as the origin of journal evaluations.

The journal impact factor was first proposed in 1963 as a spin-off from the science citation index (SCI) which was launched that year by Garfield, 1972, Garfield, 1976. Studies on performance evaluation often focus on the identification of research of the “highest quality”, “top research” or “scientific excellence”. This focus on top quality has led to the development of a whole series of bibliometric methodologies and indicators (van Leeuwen, Visser, Moed, Nederhof, & van Raan, 2000). Representative indicators are the Relative Citation Rate (Schubert, Glänzel, & Braun, 1983), the Relative Subfield Citedness (Vinkler, 1986), the Normalized Mean Citation Rate (Braun & Glänzel, 1990), the Field Citation Score (Moed, De Bruin, & van Leeuwen, 1995), the Hirsch Index (Hirsch, 2005), the Article-Count Impact Factor (Markpin et al., 2008), etc.

Methods of journal evaluation usually consist of single index evaluation or Multiple Attribute Evaluation (MAE). As research performance is multidimensional it is clear that it cannot be evaluated by a single indicator (Martin, 1996). In MAE, indicators are combined into a single index. Such an approach is widely used by statistical officers and national or international organizations to convey information in many fields. Examples of well-known MAE include the UN's Human Development Index (Sagar & Najam, 1998) and the environmental performance index produced by a joint effort of Yale, Columbia, the World Economic Forum and the Joint Research Center of European Commission (Esty et al., 2006). MAE is essentially concerned with the problem: how to evaluate and rank a finite set of alternatives in terms of a number of decision criteria. Most popular MAE methods currently used are: Weighted Sum Model (WSM), Weighted Product Model (WPM), Technique for Order Preference by Similarity to Ideal Solutions (TOPSIS), Data Envelopment Analysis (DEA) and ELECTRE. Yue and Wilson (2004) constructed a framework for the analysis of journal impact based on the principle of structural equations. Although composite indicators are frequently used for analyzing social or economic activities, they are seldom used in the scientometric literature (Vinkler, 2006).

In those evaluations, people often pay a lot of attention to the comparison of different indicators in the same year. In our opinion they, however, fail to consider the variation of indicators as time evolves. They also fail to notice how the gap in indicator values between excellent journals and ordinary journals may influence the selection of indicators. This paper explores the indicator selection issue by using the CSTPC (China Scientific and Technical Papers and Citation) database built by ISTIC (the Institute of Scientific & Technical Information of China). For more information about this database the reader is referred to (Wu et al., 2004). An improved method of Panel Data Analysis is presented for indicator selection. Then an evaluation exercise using the TOPSIS method is conducted as an illustration of our approach.