From Information to Intelligence

It is often hard to determine if a research paper is significant or not, specially if you are not involved in this area. An argurable but common criteria to evaluate the impact of an article is to study how often it is cited by other paper. To do so one may use Citeseer or Google Scholar.

The same type of metric is also used to rank resarcher. It is based on the assumption that “the more you publish, the better you are”. Various ranking index exists such as:

1. Total number of papers
2. Total number of citations
3. Average number of citations per paper
4. Average number of citations per author
5. Average number of papers per author
6. Average number of citations per year
7. Hirsch’s h-index and related parameters
8. Egghe’s g-index
9. The contemporary h-index
10. The age-weighted citation rate
11. Two variations of individual h-indice

You can even find software that compute all theses index for you. For example look at : Harzing.com - Research in International and Cross-cultural Managemen ranking software which use Google Scholar as backed.

However ratings need to be use with caution because depending on the field the number of publications have not the same order of magnitude. For example having two papers published while doing math is very impressive whereas having twenty chimical papers published is fairly common. Moreover having ten paper in stinky conferences is not better than only one but in a top conference. Citations analysis does not take into account that either.

Citation analysis for me is a good idea while used for competitive intelligence. It helps to detects important papers that have been missed during the collect or to do a preliminary analysis to find which paper to look first. However it is not a silver bullet because if a paper is cited then it is old. For buisness critical research area it is not suitable because by the time the paper start to be cited it is to late to review it.

For research evaluation, I am more reserved because such ranking tends to interfere with resarche primary goal: Citation analysis push people to publish the more they can. They have to product “articles”. It is the famous “publish or perish” syndrom. This pressure mean that reseach is no more a long term work but a short term one with a new goal : publishing. If you have to publish your quota of paper in the year, you dont have the time to explore hasardous direction that might not give results. But exploring new direction is the essence of reseach. That is why I believe that somehow this type of evaluation need to be mitigate with qualitative analysis. Rating should be considered as a hint rather than as reference or grade.