Law professors complain for good reason about law reviews: the short submission windows are ridiculous; editorial boards are always inexperienced; acceptances are skewed by letterhead bias; the actual edits are naïve and unhelpful. The list goes on. If only we had peer review, goes the lament, many of our problems would be solved.
Well, it turns out that peer review has serious problems of its own. In a post titled Our Broken Peer Review System, In One Saga, the sociologist Phillip Cohen details a two-year story of frustration with publication in his discipline. The post is pretty long, and it goes into considerable technical detail, but the gist is that he ended up submitting a short article (coauthored with a graduate student) to four journals, which he had to do successively because, unlike law reviews, they do not allow multiple submissions. Each journal sent the piece out for peer review, resulting in eight onerous rounds of “revise and resubmit.” Cohen, who is a professor at the University of Maryland, found most of the reviews unhelpful, often poorly informed or worse, and frequently contradictory. Despite a flood of input from peers, the article itself remained essentially unchanged and, according to Cohen, definitely unimproved. Here is how he put it:
From the time of our first submission to the publication date was 776 days. For 281 of those days it was in our hands, but for the other 495 days it was in the hands of editors, reviewers, and the publisher. Despite responding to 13 reviews, with a lot of tinkering, the basic result did not change from our first submission in August 2013 to our last submission in May 2015.
Cohen goes into considerable detail about the pointlessness of the peer reports. The post is very convincing (although it is a quantitative paper and I am not really able to assess it on the merits). I do know that Cohen is highly regarded in his field. I have corresponded with him a bit, and I have always found him to be thoughtful and incisive.
In any case, it is extremely helpful to see that peer review is far from a panacea. As loopy as it is, our current system has the benefits – though I would not call them virtues – of multiple submissions, generally quick responses, definitive acceptances or rejections, and the possibility of negotiating expedites and upward placements.
As always, be careful what you wish for.
In reading the article, it became abundantly clear that Professor Cohen didn't understand the methodological criticisms being leveled at the piece. While the reviewers could have been clearer, it became obvious to me very quickly that the DV was autocorrelated, which called for a time-series analysis using lagged DVs, which would require either a theoretical justification for a choice of lags or a bootstrap model to determine what lag value best fit the results. Cohen never seemed to see that.
I'm also deeply concerned by Cohen's admission that "you can't test" their explanation of their empirical findings. That strongly suggests that they haven't thought through their research design completely. I am by no means a worshipper at the altar of KKV, but they make some really great points about quantitative research design: namely, that if your theory isn't testable using data other than the stuff you used to derive your theory, you haven't engaged in science – you've engaged in data mining. To some extent, all social science does this and turns a blind eye to it – but we all at least pay lip service to the notion that the theories we create should be falsifiable, and look for ways for that to happen.
In sum, this sounds like a professor complaining that they had to work too hard for the publication, when in fact what it sounds like is they didn't have anything publishable to begin with and it took three years to find an editor desperate enough for content to publish it anyway.
Jesus Christ, I just re-read my comment and it is really offensive. I wouldn't say no to having the moderators delete everything I've posted in this thread, but in the alternative, I'll post here an apology to Professor Cohen and the editors of Sex Roles and say that even if I'm right about the substantive criticisms of your paper, which are based on my reading of the review process as described by Cohen, that's no excuse for my cavalier dismissal of your work. I'm going to walk away from the computer for a while now.
The post-publication review idea is interesting, and of course all papers are subject to post-publication review (should anyone in the field be so inclined). However, post-pub review has a couple problems. The original paper won't include citations to the later works, so a reader may not be aware of serious criticism (pre-publication review would have asked the author to deal with this criticism, thus incorporating it into the original published piece). And, it doesn't necessarily get the sort of back-and-forth that peer review ought to have. It can be less of a conversation and more the leaving of passive aggressive notes.
It may be that the artsy fartsy types have figured this out. Take something like the Best American Poetry anthologies. The poems are all published in the regular journals, and then a panel of experts picks which from the year are the best. The winners are re-published in the Best anthology. You can have a low bar for the first round of publishing, followed by a second round with serious review for the best works.
It'd also free up the reviewers to only focus on the very best, which should result in a much more meaningful dialogue than Cohen experienced.
The lag between submission and ultimate publication is a real problem in my humanities spouse's field. They were considering putting in teeth in the third year review process — i.e., "well, if this person has no publications, we reserve the right to terminate their pre-tenure contract at that point." My (tenured) spouse pointed out that even if pre-tenure folks started submitting stuff their first semester in residence, it is entirely possible that they wouldn't have anything in print at the time of the review (generally after 2.5 years). And nothing submitted after the first year would be out.
Plus there's the issue at least in her field that the blind review rule is only honored in the breach. We know someone who had his/her book tanked because the outside reviewer had just published on the same subject and didn't want to be preempted. (Fortunately the editor realized that and sent it out to another reader who gave it an honest review.)
Spouse,
One would hope that even absent publications, the pre-tenure faculty would still have completed works they could show. A review committee ought to be able to evaluate the quality of those works on their own merits. Not having been published is pretty lousy grounds for terminating a junior faculty member — not having done substantive scholarship makes a lot more sense.
Plus, and meaning as little offense to junior faculty as possible, do we really expect someone to make a meaningful contribution to their field after just a year or two in academia?
"Jesus Christ, I just re-read my comment and it is really offensive. "
Ego te absolvo.
"A review committee ought to be able to evaluate the quality of those works on their own merits."
Ha ha ha ha ha ha ha ha ha! Oh, wow, that was a good one.
In the very slight chance you weren't joking, at least in this field and in this department, unless the faculty member up for review is in a committee member's particular field (unlikely put possible), they tend to rely very heavily on journal quality and outside reviewers to assess quality. Someone up for their third year review who didn't have any publications (and probably to be fair, accepted-but-not-out-yet papers) would be seen as not being able to produce publishable material, rightly or wrongly. (They ended up not adding teeth to the third year review, mostly because clearer heads were able to convince everyone that you'd have to push the review to the fourth year to give pre-tenure faculty a fair opportunity to negotiate the peer review process, and if you did THAT, candidates who passed the third year review by the skin of their teeth really wouldn't have a fair opportunity to strengthen their tenure case given that their files would be due 1.5 years later.)
"Plus, and meaning as little offense to junior faculty as possible, do we really expect someone to make a meaningful contribution to their field after just a year or two in academia?"
Indeed.
Cohen’s saga sounded dreadful, but 776 days to from submission to publication is hardly out of the ordinary for many of us who publish in “Law and” journals. If anything, we may have an even worse peer review culture in law.
Here are the delay times for the peer-reviewed articles that I submitted pre-tenure.
1st submission:
819 days from submission to acceptance, 1617 days from submission to publication, 292 of which were in my/our hands. This was with a single journal.
2nd submission:
420 days from submission to acceptance, 799 days from submission to publication, 50 of which were in my/our hands. This was with a single journal.
3rd submission:
745 days from submission to acceptance, more than 1800 days from submission to publication, 133 of which were in my/our hands.
This was with two journals.
4th submission:
616 days from submission to acceptance, 918 days from submission to publication, 209 of which were in my/our hands.
This was with three journals.
5th submission:
103 days from submission to acceptance, 323 days from submission to publication. I can’t recall how much time it was in my/our hands, but it was fairly minimal. This was with a single journal.
6th submission:
432 days from submission to acceptance, 563 days from submission to publication, 196 of which were in my/our hands. This was with a single journal.
There was a seventh submission that was rejected relatively quickly and never published.
Law reviews have their problems as well, but at least the process is relatively efficient. Having dealt with both, I can't say that publication decisions are more or less arbitrary in peer review.
I have had experience publishing in both peer-reviewed (European) journals and non-peer-reviewed (American) journals. The dynamics of the two are just totally different. And they both have value.
American journals are run by unpaid, but very willing to work, students who are, after all, intelligent grad students. Peer reviewed journals are run by possibly overworked academics who struggle with external professors unwilling or unable to volunteer much time to peer review. The difference in end quality between the two is negligible. I have read great stuff in student journals and I have also read garbage in peer reviewed journals.
My experience has been that the American ones are harder to get placed in, altho there is the benefit of multiple submissions and leveraging up. The peer-reviewed journals tend to be more willing to see the value of the seed of an idea at an early stage and give comments that help the piece become something quite good. The American journals don't want to work with you – they just reject it if it is not near publishable already.
In the US, we value getting work out there in the marketplace to be evaluated. in Europe, the view is that journals are gatekeepers of quality, and "peer-reviewed" is a requirement for a piece to be eligible for tenure.
Rather than yearn for one or the other, I think we should value diversity in journal types. If anything, perhaps we should be looking more at a hybrid approach: faculty at the school (or external peer review) decide on submissions, but students complete the editing and cite checking process. Or something else along those lines.
So, so much to say about all of this. Let me touch on a few highlights. First, a lot of peer review is about weeding out bad work, which apparently this article wasn't. Part of the cost of weeding out bad work is that everyone else is subjected to a longer process. Second, some of the peer review process involves the authors reaching higher than they have a reasonable expectation will publish their work. By reaching high you likely extend the number of sbmisions necessary. Third, the submission process to law review may be fast … But it may also be long. I'm guessing I'm not the only person who's sent out the same paper over several submission cycles. Fourth, publication lag is unconscionably long for a lot of journals. I'm thinking about this in particular now because the article I published in American Journal of Legal Hisotry way back when I was an assistant professor took about three years from initial submission to when it appeared. I was in the situation that Academic spouse refers to. My third year review committee wondered what was going on. I could explain it to them, but it required an explanation. Fifth, God bless ssrn. At least we can get work put to the public before it appears in print….
Spouse,
Let me just make sure I'm understanding this correctly. The hiring/tenure committee members aren't necessarily capable of evaluating the quality of a junior faculty member's work if it's outside of their field of expertise? So many responses come to mind:
(1) They can ask someone in that field to sit in and advise the committee.
(2) They can just do the work. I could see there being a problem if a professor is doing quantitative analysis and the committee members just have legal backgrounds, and not math/econ/etc. But for normal legal subjects, surely they're capable of checking sources, doing a little bit of outside research, and applying their own logical reasoning to a subject to make sure everything checks out and isn't just complete arble garble. You can get an RA to do most of the grunt labor for about $15/hr.
(3) Assuming it's true that they can't actually evaluate the works that well, is the committee's position that a bunch of 3Ls can do a better job? (Or are they unaware of who's staffing these journals?)
Will,
The students on journals aren't always that willing to work. Some are super dedicated career nerds who love this stuff, but there's also plenty of students on journals just doing it for the resume line and trying to put in as little work as possible.
My 2L year I regularly turned down the cite checking that was given to me because I was too busy (1) studying for class, or (2) not cite checking. The punishment for blowing off the majority of my work? I was asked to be an executive editor 3L year, with the repeated promise of no more cite checking. I turned it down because I already had my job offer and didn't need to pad my resume more.
Maybe I was on the extreme end of disinterest, but they're certainly not all staffed with a bunch of James Harts.
Derek, the assessment of a scholar's work is not just about whether it is legally accurate and analytically solid, but also also about whether it is original (which means more than "did anyone else say this" but could include whether it follows even indirectly from work someone else did or is analogous to work someone else did on a related issue versus whether it represents a paradigm shift) and perhaps more importantly whether it makes a contribution to the field. Making those last two judgments requires depth in the field. A law professor in an unrelated field is almost as poorly equipped to make those calls as a third year law student.
Just about on cue, a post like this pops up annually apparently to demonstrate that the law review process is superior or has its virtues. But one scholar's gripe about one particular article hardly makes the case for a student-edited process which limits submissions to once, or twice a year, and chooses articles within 30 days sometimes much sooner of submission by students who often know nothing about the subject they are reading. Law review articles do tend to come out sooner than peer-reviewed articles but they are not the better for it, which is evident by the fact that no one else uses such a silly process and law review scholarship is rarely cited outside of law (in part because the articles are too long, which is due to uninformed student editors who require lengthy windups in every piece). Peer review is far from perfect but it is much more perfect than student-run law reviews.
Hey Lubet, the plural of anecdote is not data …
I can speak to this from personal experience in 2 contexts. I have been well-published in some very prominent law journals USA student edited and a few peer reviewed ones. The USA system is superior let me explain: Once I submitted an article to a UK based peer review journal and within an hour I received a rejection from an editor explaining the paper was "not up to par with the journal's standards". Funny, I then sent it out and it was accepted within days by a very prestigious US law school journal. Obviously, the peer review bloke didnt even send it to peer review he was the gatekeeper and I read his bio he had written on the topic and advocated the opposite approach. So his rejection I believe had everything to do with personal prejudice. Two, I on a different occasion I sent a paper to a peer review journal and it received accolades (ie no revision needed) by one expert. The second expert wrote me that on some esoteric point it needed revision or he/she could not recommend publication. I re edited several times never to his/her satisfaction UNTIL I finally realized the reviewer had probably written on the topic and wanted me to cite to their work. So I added some cites to publications to the last edit and lo and behold the reviewer THEN recommended it!!! So he/she wanted me to cite to them and this lack of citation held back their recommendation/. What a stupid system.
A tiny little point that follows up on Peer Review Sucks' comments about citation.
I reviewed a piece for a peer-reviewed journal that DID cite my work. But it was terrible. But it cited my work… Anyway, I suggested the journal decline the piece, but it was slightly harder to do after the author(s) had shown such good judgment in citing my work.