Correlation between reference managers and the WoS

Even though web citations have been a part of our lives for several years now, the correlation between "traditional" citations and web resources like Mendeley, CiteULike, blog networks, etc. hasn't been thoroughly studied yet, and any new research in the field is very interesting (to me, anyway).

The new paper was published at Scientometrics by Li, Thelwall (still one of my dissertation advisors) and Giustini. They focused on the correlation between user count - the number of users who save a particular paper - and WoS and Google Scholar citations.

The researchers extracted from WoS all the Nature and Science research articles that were published in 2007 and their references. They ended up with 793 Nature and 820 Science articles, or 1,613 articles overall (not including references, of course). Then, they searched CiteULike for those articles' titles and number of citations, as well as for their user count in Mendeley. They also collected the same data from Google Scholar. It's important to note that Mendeley had 32.9 million articles indexed while CiteULike had only 3.5 at the time of the study.

Google Scholar's mean and median number of citations were higher than in WoS (not surprising; If you want better citation numbers, always use GS). They found that despite Mendeley being "younger" than CiteULike (launched in 2004 and 2008 respectively), CiteULike had only about two-thirds of the sample articles saved, while Mendeley had about 92%.

Spearman correlations between citations in GS and WoS were high in this research (0.957 for Nature and 0.931 for Science). The correlations between Mendeley's user count and the citations in GS and WoS were also rather good (0.559 and o.592 for WoS and GS respectively for Nature, 0.540 and 0.603 for Science). CiteULike had far weaker correlations: 0.366 with WoS and 0.396 with GS for Nature, 0.304 with WoS and 0.381 with GS for Science.

Limitations

The authors remind us that correlation isn't causation, saying they can't conclude a casual relationship based on correlations between two data sources. Therefore, it can't be determined for sure whether there is a connection between a high user count and a high number of citations. Only Nature and Science were studied, so it can very well be that the results aren't true for other journals. Also, group-saved and single-user saved references were given the same weight. The number of saved references in Mendeley and CiteULike is much smaller than in the WoS counts and therefore the results might be less reliable.

The authors speculate that user count may represent a more accurate scientific impact of articles, and take note that one can measure the impact of all sorts of resources in online reference managers, unlike in the limited bibliographic indexes.

I think it could be reference managers don't always reflect readership: one could save a reference and forget about it all together later (so many articles, so little time...). On the other hand, citation counts might suffer from the same problem, as many scientists use a "rolling citation" from other articles citing an earlier article, without actually having read the article themselves.

Priem et al. also presented lately a study about web citations and WoS citations, based on data from the seven PLoS journals, but I think I'll wait for the journal article to cover it in the blog.

Li, X., Thelwall, M., & Giustini, D. (2011). Validating online reference managers for scholarly impact measurement Scientometrics DOI: 10.1007/s11192-011-0580-x

You're just a number: introduction to the h-index

Measuring a single scientist's output has always been problematic. Why? First, in order for the statistics to be reliable, the scientist has to produce a considerable publication output and get cited. That takes time. Second, measures like research productivity, number of publications and citations don't always correlates. Measuring the output of journals and universities has been far more reliable than measuring that of one person.

Suggested by physicist Jorge Hirsch, h-index (2005) offers an attractive way of quantifying one's scientific output as a single number. The index is defined as:

“A scientist has index h if h of his or her N_p papers have at least h citations each and the other (N_p − h) papers have ≤ h citations each” (Hirsch, 2005).

So, if a scientist published at least ten papers, which each were cited at least ten times, her h-index is ten. A zero h-index, on the other hand, says that the scientist perhaps published papers, but is yet to have an actual impact.

The h-index is attractive because it takes into account both the number of publications and the number of citations. It isn't phased by "one hit wonders", but favors a body of work that each of its components has at least a certain impact (citations).

Problems and disadvantages

Which database to use? Different databases cover different journals, conferences, etc. Web of Science, for example, has better coverage of STEM than of the humanities, which tend to publish books rather than papers. Using Google Scholar will likely inflate the h-index.

Which field are you in? Larger fields mean a larger potential for citations, resulting in a higher h-index.

You aren't a number! (Or at least, not just *one* number). Reducing scientists to a single number ignores other factors, such as their teaching skills and ability to collaborate. Can an entire career really be described as a single number?

Source: PhD Comics

The age factor: The older the scientist gets, the longer she had to publish and get cited. Younger scientists are at disadvantage with the h-index.

Relevance: Since the h-index doesn't decrease, it can't tell whether a scientist is still active and/or where her work is still relevant for others in her field.

Since the h-index is a single number, scientists with the same h-index can have very different numbers of papers and citations. In the following table, scientist A and scientists B have the same h-index, but scientist A has far more citations in the overall raw calculation.

Source: Bornmann & Daniel, 2008

Because of the h-index many problems, offering new corrections to it, or coming up with other indices altogether is the official new sport for bibliometricians. The new indices are supposed to offer a better way to make decisions about promotions and grants, but despite all the efforts, it seems that the way to the promised tenure will continue to be paved with peer evaluation.

Bornmann, L., & Daniel, H. (2007). What do we know about the h-index? Journal of the American Society for Information Science and Technology, 58 (9), 1381-1385 DOI:10.1002/asi.20609

Bornmann, L., & Daniel, H. (2008). The state of h index research. Is the h index the ideal way to measure research performance? EMBO reports, 10 (1), 2-6 DOI: 10.1038/embor.2008.233

Hirsch, J. (2005). An index to quantify an individual's scientific research output Proceedings of the National Academy of Sciences, 102 (46), 16569-16572 DOI: 10.1073/pnas.0507655102

The Matthew Effect Strikes Again

The new Bornmann, de Moya Anegón and Leydesdorff paper, published in PLOS ONE, shows that highly cited papers tend to reference other highly cited papers more often. That is true especially for the life science and health science disciplines. Ms. Corbyn from Nature News saved me the need to summarize the paper by writing an excellent article about it.

Based on their findings, Bornmann et al. suggested to concentrate funding on already highly-cited researchers and research groups ("A concentration of resources on these elite structures seems to be practical especially for the life sciences and health sciences"). This is already happening, to some extent, but if the authors' offer is to be accepted, I suspect highly-cited authors will forever remain at the top, simply because they will get most of the funding.

Now, citations don't exist in a social vacuum; despite the tendency to see them as "objective" representations of papers which have influenced other scholarly works, they are affected by factors like the paper's publishing journal, personal connections, coverage of the paper in the mass media, and more. Bornmann et al.'s recommendation might cause funding agencies to end up giving their money to the most popular, connected researchers, who are already well-established in their discipline.

Bornmann, L., de Moya Anegón, F., & Leydesdorff, L. (2010). Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis PLOS ONE, 5 (10) : 10.1371/journal.pone.0013327