Peer Review vs. Search Engine Places - Quality of Information in a Web 2.0 Sources

Interesting article from EDUCAUSE Review, vol. 41, no. 6 (November/December 2006): 12–13 - Scholarly Reputations: Who's Got Buzz? that discusses whether:

scholars would choose (1) to have their work published in the premier journal in their field or (2) to have that work regularly come up on the first screen in an appropriate Google search

Having gone through a PhD process and having been in academia for 11 years where you needed to "publish" - this is a really interesting question to me. I often argued with (older) colleagues at the university that it would be better to have open, fast publishing sources than the very slow, peer review process associated with most publications. But what really caught my eye was the statement:

For information-seekers using their favorite search engine, quality is simple to define. If a result displayed on the first screen seems to be clearly objective in perspective, if it is not part of some commercial scheme (e.g., a link to buy this book at zzz.com), and if it is immediately available for full-file use, then it is deemed to be of high quality. Moreover, because the seeker has quickly been successful in obtaining a work of high quality, there is a transferal of quality to the work's authors, who receive a high "expert" ranking. The effect may not be rational but it is increasingly true, regardless of whether the work is otherwise associated with any peer-reviewed process. New graduate students in physics can be taught to begin research with review articles such as those in the highly respected Reviews of Modern Physics. Yet it is increasingly likely that they will first search Google; the article that shows up on the first page of results will be a winner no matter what its published pedigree.

My gut tells me that this is pretty accurate which argues that its possibly more important to be high up in Google search results than to be published in peer reviewed publications. Of course sometimes, both things happen such as in the case of Stephen Downes' article on eLearning 2.0 which comes up first in the Google search of eLearning 2.0. Of course this is a self-serving example, because looking at the list you'll see that I'm listed second and Stephen comes up again third. I'm grateful to be listed second but a few other scholarly articles on the subject are quite buried.

It's interesting that being ranked second and in-between two things by Stephen (who coined the term), definitely lends a mark of quality and high expert ranking for me on the topic. Is that right? Is it appropriate? Well as the authors point out, that's almost no longer the point - it simply is what it is.

But that's not always the case, even in a Web 2.0 world and even with the same term. Consider what happened around the definition of eLearning 2.0 on Wikipedia. I contributed the original definition using a combination of writing from Stephen, myself and a few other folks (mostly bloggers). I also included links to articles and posts that I thought were relevant - and who influenced the definition. About 3 months later, a Wikipedian came along and edited the article and selectively deleted links to some of the sources I had cited but not to others. Okay, I admit it, I'm a bit sore because links to my posts were deleted. The comment I received on how it was decided was that my posts were in a blog not in a peer-reviewed publication. I almost responded, "but there are other links to blog posts" but luckily realized that might mean deleting links to other "good sources" (in my opinion) and loss of all links to sources that are critical of the term. Don't worry though - I have an article coming out at some point and then I can add another link to something that has more credibility than my blog. :)

The point of this is that while the Educause article says:

In the Web 2.0 world, the quality of a resource is determined by the intensity of its use. What is rated as "the best" is what is used the most and what shows up first.

that's not quite right given that within some Web 2.0 domains there is still a form of review even if it shows up second. Ratings are often used. And all of this together is a form of review. What Web 2.0 does is distribute "review" more broadly and makes it much, much faster. The authors of the article clearly favor old-school peer review's quality filter. And, while I guess I'd agree that maybe not every vote should be equal, I don't necessarily agree that old-school peer-review is all that great. Maybe it's old scars in how peer review processes often work (political, old-boy network).

The real point is that even within a distributed, fast review model of Web 2.0, there are examples of old-school type review such as in the case of Wikipedia. And that felt even worse than the journal peer review process because the expectation is equal treatment.