December 11th, 2010

"Smart crowdsourcing peer reviews"

I feel sorta bad for poking at this phrase, because now I can't stop thinking about it. It is very easy to stop thinking about someone who proposes an R&D group at a University Press -- that's just silly. But "Smart crowdsourcing peer reviews" is tough to let be.

My first thought brought me back to the very mildly interesting Bloomberg news interview with Reid Hoffman that sent me down this whole academic publishing rathole in the first place: maybe LinkedIn could help connect potential, um, manuscripts just seems _really_ wrong in this context, e-books for publication to peer reviewers. Then I got to thinking, today, about the way TUG (Timeshare Users Group -- see, this is what it's like inside my head) solicits reviews from its consumer base.

TUG is a bunch of people who own timeshares and visit timeshares. They are interested in learning about other timeshares, and they know about some timeshares directly. Thus, every participate in the users group is potentially a producer of reviews and a consumer of reviews, and many are both. Same thing in academia. Fresh reviews are better than old reviews (not totally clear how that translates to e-publishing academic books, but hey, nothing is ever exact); more reviews are better than fewer. IIRC, TUG doesn't have any particular reputation system for assessing reviewers, or for assessing the utility of a particular review (Amazon does have the latter; eBay has a parallel for the former).

You could take all this stuff and readily imagine a replacement for editorial staff and/or reviewers in general (such as at Choice), in which you submit a text (has to satisfy format requirements, but you can contract out that part). There could be compensation (honoraria) for the first few reviewers, who might be required to meet particular requirements (registered with the system as having particular credentials in particular areas of expertise). Separate from the "peer review" bar being met, you could also have general (uncompensated, probably) reviews.

It's pretty easy to imagine that some really fast reader with the right credentials in enough fields might decide they could sweatshop themselves and make a living just off of reviews. Whether that's a good thing or not is entirely debatable; some cases might require some investigation to ensure that reviewer ethics are being met. That, in turn, would lead to an entertaining debate on just what reviewer ethics are or should be, exposing a sordid underbelly of academia that could do with a little sunshine. And it's not like we don't have professional readers already, anyway, right?

If you're not asking, why not just do a wiki with credentials, you probably should be.

ETA: Similar ideas used for doing translation work.

http://www.insidefacebook.com/2010/08/09/cloudcrowd-crowdsourcing-peer/

On facebook, no less.

ETAYA: Less sweatshoppy, more relevant to the problem in question:

http://www.nytimes.com/2010/08/24/arts/24peer.html

The commenters were selected to contribute and they signed their work -- and there were other reviewers as well.

The article references this:

http://www.nytimes.com/2010/08/17/science/17proof.htm

How did I miss that!!! I could have had a whole week in August being totally geeky and insufferable following the discussion! Yes, I know. No one else cares.

ETA last time I swear: After digging around in wikipedia for a while and reading about Sokal and the French pseudo-scientist twins and publication subvention and a few other odds and ends, I have concluded that attempting to sweatshop the peer review part of the process is probably missing the point. Totally technically feasible, won't actually help anyone.

underpants gnomes, or, I love my life, etc. redux

I've been wandering around reading about university presses, partly for the humor, but also because I find it perversely entertaining to watch the sausage that I Love So Much be made. Yesterday, when I was busy making fun of some of the blue-sky types, I was concerned that sanity might not be prevailing in university presses. However, once I stumbled upon the trade association, I decided sanity _was_ prevailing. This slide show was particularly heartening.

http://aaupblog.aaupnet.org/?p=198

http://aaupnet.org/programs/annualmeeting/2010/pratt.pdf

Basically, some New World Archaeologists were trying to figure out how to do digital monographs. They had a grant and they have a variety of participating university presses. Pratt uses the underpants gnomes joke to show the disconnect between what they were trying to accomplish (digital monograph) and where they wanted to go (profit). As a result of this disconnect, they switched focus from technical innovation to business innovation (aha!), and learned a few things about underpants. Slide 7 is particularly good:

The people (archaeologists) wanted this done
They had to limit the juicy multimedia goodness ("We are going to have to set strict
parameters regarding enhancements and how they are incorporated")
Senior scholars not junior would have to be leading the way

Really very shockingly sensible of these people.

They figure if they can come up with a channel agnostic (works on iPad or kindle or wherever) and XML based publishing workflow, they'll be in business, especially if it turns out to work for disciplines other than New World Archaeology.

All good stuff. Sanity may yet prevail. Slowly.