When is webometrics most useful?

Like many terms in Information Science (including 'Information Science' itself) the term 'webometrics' is pretty vague. Björneborn and Ingwersen (2004) defined webometrics as "the study of the quantitative aspects of the construction and use of information resources, structures and technologies on the Web drawing on bibliometric and informetric approaches." I guess this definition will have to do for the time being.

Thelwall*, Klitkou, Verbeek, Stuart and Vincent (2010) set out to find in which fields webometrics is most effective. The result is quite a long paper, that I'm going to be very general about its conclusions. As expected, webometrics doesn't have the same effectiveness in every field. It is at its best with emerging and/or "hot" fields. That is because web publication is easier and faster than publication in traditional scientific outlets, and researchers can publish ongoing results with little delay.

In some disciplines plenty of their products aren't regularly published in journals (social sciences, humanities, applied fields, etc.) and therefore aren't as well-covered by bibliometrical databases as disciplines with a journal-publishing culture. Bibliometrics is also bound to have a poor coverage of multidisciplinary fields, because their outputs are published in various outlets and are often cited in different manners.

In general, webometrics analysis gives better results in fields with standards and/or norms for web publishing, but the results might not be reliable in fields where a small number of research groups and their projects (databases, web portals and so on) have a disproportional web presence.

Collaborations are often better caught in webometric analysis, since not all collaborative works have "official" outputs.

Webometrics works better for smaller fields. It's harder to get a complete picture of large fields with current methods.

Last but not least: webometric analysis is usually faster and cheaper than bibliometric one.

Thelwall and his colleagues concluded that "whilst webometrics is still inferior to bibliometrics for most purposes it seems that it has advantages for some types of field, particularly new, small fields, and can deliver policy-relevant (process) indicators to promote effective collaboration and communication," (in short: use with caution).

Appropriate disclosure: Prof. Thelwall is one of my dissertation advisors . My favorite from his long list of achievements is that he managed to publish a serious research paper including a YouTube cat video.

Thelwall, M., Klitkou, A., Verbeek, A., Stuart, D., & Vincent, C. (2010). Policy-relevant Webometrics for individual scientific fields Journal of the American Society for Information Science and Technology, 61 (7), 1464-1475 DOI: 10.1002/asi.21345

Don't say you found aliens (unless you actually have)

Unlike with health and medicine press releases (Woloshin and Schwartz have a few good papers about the matter) I haven't seen much research about other scientific press release. That's why I was glad to find the paper "Credibility of science communication: An exploratory study of astronomy press releases" by Nielsen et al. (2007).

They conducted 11 in-depth interviews with journalists, scientists and public information officers, and came up with several conclusions regarding the accuracy and credibility of astronomy press releases.

• Credibility was defined by the interviewees as "being honest and doing your homework." Hype was defined as overstating the importance of results in order to increase visibility.

Credibility problems with press releases

Problems were usually caused by either the press release trying to make the issuing institution look better, or trying to make other institutions look worse.

1. The level of communication effort: finding some well-known person to tell the media how much the research described in the press release is significant to science.
2. The wording of the press release: even if it is possible you found alien life, keep that question mark at the end of "Alien life found?".
3. Dictating the timing of a press release: we just happened to discover something really important, just in time for the annual budget meeting! Other time-related sins are publishing the press release before the peer-review paper is out, and timing the release to screw up the competition's own press release or event.
4. Omission of reference to other scientists' work: this isn't the 17th century. It is hardly likely you did everything by yourself, or haven't built on some previous research.
5. Unjust comparison with other facilities.

Good ways to avoid lack of credibility (which can be in science, unlike in politics, problematic) is to have internal referees to the press release before its publication. Also, the importance of a peer-reviewed paper backing the press release can't be overstated.

Despited everything said here, the authors' overall conclusion is that "...credibility problems for astronomy press release do not exist, though examples certainly exist."

I find this conclusion very encouraging.

Nielsen, L. H., Torpe Jørgensen, N., Jantzen, K., & Christensen, L. L. (2007). Credibility of science communication: An exploratory study of astronomy press releases Proceedings from the IAU/National Observatory of Athens/ESA/ESO Conference, Athens, Greece.

The Volokh, the slashdot and the NYT effects

Back in 2007, Paul Ohm, a law professor in the University of Colorado law school, guest-blogged in a popular law blog called The Volokh Conspiracy. He guest-blogged for one week about two of his papers: "The Analog Hole and the Price of Music: An Empirical Study" and "The Myth of the Superuser: Fear, Risk, and Harm Online." Being more computer-savvy than the average law professor (he has a B.Sc. in Computer Science) he wrote a script which checked the number of abstract views and downloads of his papers from the Social Science Research Network (SSRN). Both have gone up. Then his posts were linked to by Slashdot, and the numbers went up even more (graph taken from Professor Ohm's paper).

Currently, "Superuser" has been downloaded 1,434 times and its abstract viewed 8,859 times. The "Analog" paper has been downloaded 412 times and its abstract viewed 2,942 times. According to Google Scholar, "Superuser" has been cited 29 times and "Analog" only five, so there is a certain correlation (which I admit I didn't calculate) between downloads and views in SSRN and citations (assuming GS' citation count is more-or-less accurate. Do that as your own risk).

But that's not the entire story. Looking through Ohm's other papers, I've noticed that this paper: "Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization" has been downloaded 5,827 times and its abstract viewed 22,528 times, even though it was only published last year. A quick Google search brought up the possible reason: the paper has been mentioned in a New York Times blog, as well as in this site. It has also been cited at least 4 times (GS is a bit confusing because it presents two entries for this paper).Naturally, there are many more variants at work here (for example, the subject of the paper) but I think this is a demonstration of the power of central, as opposed to niche blogs and sites.

Paul Ohm (2007). Do Blogs Influence SSRN Downloads? Empirically Testing the Volokh and Slashdot Effects U of Colorado Law Legal Studies Research Paper

Scientometrics 2.0, part I

Today we're going to discuss this paper, by Priem and Hemminger (2010), dealing with Scientometrics (a general name for Bibliometrics, Webometrics, Influmetrics, and all sorts of other metrics regarding scientific activity). But, before we start, I want to protest First Monday's terrible references format. Sure, it looks fine in-text, with the last name and the year, but at the reference list the references start with the first letter of the author's first name! How am I supposed to know the author's first name? Only Darwin knows.

Anyway, Priem and Hemminger offer a state-of-the-art review of Web 2.0 tools that can be mined for scholarly data. They suggest seven categories of tools, of which I'm only going to mention four in this post.

1. Microblogging - Microblogging means Twitter. Twitter is used by scientists, among others, to discuss papers and conferences. From my experience, many people tweet from conferences under the conference hashtag.

2. Social Bookmarking - Social bookmarking services like Delicious, Connotea and CiteULike can be mined and show scientific trends by pointing out the popular papers bookmarked and popular tags.

3. Wikipedia - Wikipedia is popular among students and faculty as a starting point for basic knowledge. For other web users, Wikipedia is often their only knowledge source. So, papers cited in Wikipedia are more likely to have public impact. Indeed, the JCR and Wikipedia citations correlate well.

4. Blogging - By now, blogs are well- established in the web culture. Many scientists, Fields Medalists included, maintain scholarly blogs. When discussing academic papers, those scientists often cite their sources in traditional manner. Excellent examples are posts aggregated by the Research Blogging service.

That's it for today. The next part will discuss other categories of Web 2.0 services with possible scholarly use.