What Good is Legal Scholarship?

Michael,

I'm not sure I'm with you all the way through, but your last paragraph is really fantastic. Great point.

Anon

"Indeed, the notion that scholarship might be of interest to only other scholars is viewed with derision."

I think what the article views with derision is the idea that scholarship that is of interest to only other scholars should receive the level of subsidization that it does.

"The reality is that citation rates vary widely by discipline. Engineering sciences are right in the middle; maybe we should tell the engineers they should stop seeking all those government grants because no one ever cites their work."

"I've read hundreds (thousands?) of computer science articles. Many fascinating, many containing novel applications. Most did not disclose something that became commercial software. Perhaps they should just stop trying."

I know this was tongue in cheek, but I don't really see what you're getting at. People outside academia find scientific inquiry worth paying for. You can't say that about legal scholarship. When faculty in the sciences/engineering start expecting their students to subsidize their research you might have a point.

(http://phys.org/news/2014-12-scientific-citation-nobel-laureates-papers.html)

Something you might find interesting.

"People outside academia find scientific inquiry worth paying for. You can't say that about legal scholarship."

This statement might surprise the legal scholars who have received the MacArthur genius grant (among many, many others who have received funding from a variety of sources.)

Seems like you wanted to say something but really could not think of anything. As I read it, they concede that articles may tell us something about the world but question whether everything law professors tell us about the world is worth it. Did you miss that? Plus, as another commentator as pointed out, they have underestimated the cost. More likely, your post makes their point. Good day, sir!

"This statement might surprise the legal scholars who have received the MacArthur genius grant (among many, many others who have received funding from a variety of sources.)"

Fine. That isn't absolutely true. But the point is that external funding of legal scholarship is, at best, a drop in the bucket (compared to the $250k/yr engineering professors bring in).

Do external sources of funding account for more than a couple percent of the support for legal scholarship?

your point on marginal costs needs an explanation. They are all marginal in the long run so you must mean an article by article analysis in which the marginal cost and benefits of an article are compared. If so, the article's conclusion that there is a surplus is correct. Actually quite obvious. What is also obvious is that law profs will not self regulate.

Wrt to John jacob's comment. I am sure you are right. The logic of using an exception to disprove a inconvenient truth is typical law prof reasoning. Think of the global warming doubters who note that it was really cold today.

"Indeed, it is odd that the article first argues that citations are not a good way to measure the value of scholarship and then uses lack of citations to argue that scholarship is valueless."

One can easily say that citations in academic journals are a poor proxy for value while also believing that citations in judicial opinions are an excellent proxy for value.

"First, I believe they understate the value of articles read but not cited. There are many uses of articles by professors and lawyers that never wind up being cited anywhere, though this is surely subject matter based. As the draft notes, SSRN downloads (and abstract views) far outstrip citations."

Not sure this is an argument the legal academy will want to advance. For starters, SSRN downloads are themselves not a great proxy. There are other ways to read articles, not everyone distributes their SSRN link as their main way of getting it out, and then not every article which is downloaded ends up getting read, and not every article that is read is thought well of by the reader (you could have the 50 Shades of legal papers, something widely read, and just as widely condemned as junk).

But, the bigger reason why the academy might not want to say "you have to look at how many people read these things!" is that not that many people read these things. With not quite 1500 total downloads over 3 papers (far less than Risch's 12,800!), I'm at roughly the 93rd percentile for all authors on SSRN. A lot of you lugheads need to catch up. I'm not going to name names, but two TFL contributors who have been at it for a decade haven't even cracked the 1000 download mark.

And of course, if we started using SSRN downloads more seriously in evaluating the value of papers, we'd just see a lot more gamesmanship. If you teach a 100 person 1L course each semester, you assign a couple papers written by you and your friends, and suddenly you're at the top of the heap. Don't even need to actually discuss the papers in class.

But despite all the problems with the various proxies, they seem to keep coming up with the same conclusion: while some scholarship is very valuable, the vast majority of it is of marginal or no value. Do we really want to sink half a Billion dollars (maybe a bit more) into that every year? Or to put it another way, would we be better spending our money on something else, such as slashing tuition by a quarter or dramatically reducing the justice gap?

All – these are thoughtful comments. So much so that I'll probably address them in a followup post in the next couple of days!

In brief answer to Derek Tokaz, though, I don't mean to say we should judge value by SSRN downloads and abstract views. I'm just saying that influence likely extends beyond the citations. Even your 1500 downloads is likely more than you've been cited, no? That's my only point, not that SSRN should substitute.

Now, whether THAT'S enough to "sink a half a Billion dollars" into is another story, which I'll get into later.

Michael,

Of course I have more downloads than citations. I'd hope that anyone citing an article would have at least downloaded it! I've got something like 12 or 13 citations. No clue if that's a particularly good number or not. (And I'll note SSRN is poor at tracking citations. I counted mine manually.)

As for not judging articles by SSRN downloads, sure. But, judging them by how widely read they are? That seems more reasonable, and if it is, then SSRN downloads may be a good proxy. And if we're not going to use SSRN, what evidence is there that anyone is reading these things?

It seems like the current thinking of law schools is "We know that some of these things are truly of great value, and we can't prove for certain that the rest are not, so let's throw as much money at it as we can." Personally, I think the burden of proof ought to run the other way. Want to only teach part-time because you're doing scholarship? Demonstrate its value.

Yes, we are all such philistines for suggesting that legal scholarship should have some demonstrable use, especially for those who put themselves in massive debt to attend law schools.

Law is unique among graduate programs that aren't medical or dental school for its cost of attendance. For example, your school, the Villanova University School of Law, had a total cost of attendance of $237,923 for its graduating class of 2013. Maybe it could charge less to successive classes by subtracting from tuition the amount that would ostensibly be used to pay for professors' non-instructional time researching, and replace it with individual professors' Kickstarters or tip jars scattered throughout the law library. And then, once tuition subsides to a point where it's only 150% of what law schools cost to attend during 1980, perhaps us philistines will be far more reticent about our criticism of your impractical but fiercely beautiful adventures of the mind and human spirit.

You, like the other commenters, are showing your priors. I'll discuss this later.

But that said, you can also believe that SSRN reads are ALSO not enough to show influence. I don't have evidence of that. My point is only that citation counts, as an absolute measure of influence, fall short. I don't see that as contestable.

Some interesting excerpts from “Citation Statistics: A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS)," published Jun. 12, 2008:

“The validity of statistics such as the impact factor . . . is neither well understood nor well studied. The connection of these statistics with research quality is sometimes established on the basis of ‘experience.’ The justification for relying on them is that they are ‘readily available.’ . . . [C]itation data provide only a limited and incomplete view of research quality, and the statistics derived from citation data are sometimes poorly understood and misused. Research is too important to measure its value with only a single coarse tool . . . .

This much isn't new: People have been assessing research for many years. What is new, however, is the notion that good assessment must be ‘simple and objective,’ and that this can be achieved by relying primarily on metrics (statistics) derived from citation data rather than a variety of methods . . . . [T]his faith in the accuracy, independence, and efficacy of metrics is misplaced.

. . .

Many writers have pointed out that one should not judge the academic worth of a journal using citation data alone, and the present authors very much agree. In addition to this general observation, the impact factor has been criticized for other reasons as well. (See  [Seglen 1997], [Amin‐Mabe 2000], [Monastersky 2005], [Ewing 2006], [Adler 2007], and [Hall 2007].)

. . .

Once one realizes that it makes no sense to substitute the impact factor for individual article citation counts, it follows that it makes no sense to use the impact factor to evaluate the authors of those articles, the programs in which they work, and (most certainly) the disciplines they represent. The impact factor and averages in general are too crude to make sensible comparisons of this sort without more information.  

. . .

Life sciences (6.2)
Neuroscience (4.5)
Clinical Medicine (3.3)
Pharmacology (3.1)
Physics (3)
Chemistry (2.9)
Earth Sciences (2.3)
Environmental sciences (2.2)
Biological sciences (2.1)
Materials science (1.2)
Social science (1.1)
Mathematics/Computer (.9)

(“[T]he journal impact factor is computed by calculating the average number of citations to articles in the journal during the preceding two years from all articles published in
that given year (in the particular collection of journals indexed by Thomson Scientific). If the impact factor of a journal is 1.5 in 2007, it means that on average articles published during 2005 and 2006 were cited 1.5 times by articles in the collection of all indexed journals published in 2007.”)

SSRN downloads are a bad measure of scholarly impact for scholarship for a another reason: the process of counting them inhibits reading. SSRN's lack of direct PDF links and its download integrity system drive down readership.

A deeper point here is that to the extent scholarly impact depends on reaching a broad audience, academics have an obligation to make their works readily accessible — an obligation many legal academics discharge indifferently at best. Journals are helping now by posting their articles online, although their archiving is often quite poor. But this process further reduces SSRN download counts; readers of recently published work find it elsewhere online. That's a further reason to take SSRN download counts with the idiomatic grain of salt.

James,

I'd argue that SSRN's lack of direct PDF links increases the value of its numbers as a proxy.

Direct linking is easily abused to boost statistics. The two-step process helps to ensure that the article is only downloaded by people who know what they're downloading.

That said, I agree that one serious flaw is that SSRN is not the only source for the articles, and we do not even know if it is the most popular source. But again, this is where the burden of proof comes into play. It shouldn't be enough to say "Well, maybe people are downloading it elsewhere." It should be on the professors (or the academy as a whole) to demonstrate that the articles are actually being read through other means.

The authors cite the Chen article for the fact that ssrn downloads are slanted toward the highest ranking schools. Does anyone actually download or not based on the prestige of the author?

Some data: I have roughly ten times as many downloads through BePress Selected Works as I do through SSRN. The numbers aren't consistently proportional; some articles are far more than ten times as popular on Selected Works and some are far less. I don't know how representative my experience is, but it strikes me as a strong case for open access.

That was my original point: open access is better for the world than good download counting. I don't have anything against drawing appropriate inferences from download counts where we have them. But the pursuit of the status that comes with high SSRN download counts has ironically led law professors to put barriers in front of their work, reducing their impact in the world. Institutional repositories and law-review websites are both good developments.

James,

"But the pursuit of the status that comes with high SSRN download counts has ironically led law professors to put barriers in front of their work, reducing their impact in the world."

Can you explain what this barrier is? As far as I can tell, SSRN is free for anyone to access.

There are a few issues but the worst is that The suspicious download system can get caught in infinite loops. You bounce between "download this paper" and "download anonymously" (or logging in) forever.

John Thompson –

I just fished your comment out of spam. Philistines is your word, not mine. I'm merely saying that your position is colored, obviously, by a prior normative view about what law school professors should be doing. I think you're wrong about that, but I don't expect you to agree with me.

But while you are law school tuition (and Villanova specifically) bashing, let me just note that factually you are off. First, law schools are not singularly expensive. Undergrad tuition is a fortune, business school, engineering school, and other tuitions are not cheap. You also have no idea how much of the tuition in each of those places is covered by financial aid, a common failing of reporting raw tuition numbers.

Second, as to Villanova specifically, our students had average indebtedness that ranked about 70. That's not the best, but it's 69 spots better than worst.

Michael,

We actually do have quite a bit of information about how much of the tuition in each place is covered by financial aid.

In 2013 (most recent data), at Villanova 35% of students received no scholarship. Another 18% received a scholarship for less than half of tuition. 19% got full or better. The median scholarship was $25,000, $5,000 more than the previous year. 7% more students got a full ride than the previous year. But, I think that's all beside the point. Law school is expensive. It doesn't get less expensive just because other programs are also expensive. And even with a $25,000 a year scholarship law school is still expensive.

Scholarship is a huge expense, and it's one that's being paid for primarily by tuition dollars and much (if not most) of that money is debt financed. At the same time, the vast majority of legal scholarship has no demonstrable value — it's value is merely speculative. Now that's not an immediate knockout punch, lots of things have value that's hard to quantify or prove. However, the explanations for the value of legal scholarship are pretty dubious and the proxies for value points towards the scholarship being closer to worthless than worthy.

Just to clarify, the scholarship in the second paragraph is the financial aid type. The scholarship in the third paragraph is the academic research type.

Interesting – I didn't know that scholarship data was published.

You are right that it is besides the point. Either you believe that legal scholarship is justified ONLY IF it has some external value or you don't. I'm going to blog on that, hopefully today.

Michael,

One thing I hope you'll address is how the scholarship is paid for. Personally, I enjoy reading and writing scholarship, and it can be very useful beyond how many citations it gets in court documents or other law journals. But the question I consistently ask is not whether the scholarship has value, but whether it's something that should be paid for by students. And if so, how much? When a legal education costs as much as it does, you must consider the source of revenue and the impact on that source in addition to the amount. A serious discussion on this point goes beyond saying schools ought to (and do) look elsewhere to fund important work.

Michael,

It sounds like you're alluding to the argument that scholarship has value because it makes professors better at teaching (keeps them up to date, keeps their wits sharp, whatever). That would certainly address all the concerns about lack of readership, meaningful citations, and impact. It would also speak to the funding issue, as it makes sense for schools to spend a lot on professional development (though how much is still a question).

If that is the direction your post goes in, I hope you'll address alternatives, such as professors engaging in legal practice (either through a clinic, their own practice, or as a sort of adjunct attorney at a firm), attending CLEs and the like, and simply teaching more classes.

It would also be good if there was some way to verify that scholarship improves a professor's ability to teach, such as finding a correlation between research and teaching evaluations.

There can be 'good' legal scholarship. Here is the formula for writing it:

1. Only write a federal conlaw (especially first amendment, fourth amendment, equal protection, or due process) article if you have a novel insight that you can't find elsewhere in print.

2. Articles that synthesize or point out conflicts in an area of law (no matter how trivial or boring) are very helpful to lawyers, students, judges and law professors.

3. There are many, many areas under served by legal scholarship. These primarily are areas of state regulatory or state constitutional or state statutory law. If you are willing to slum it in the laboratories of democracy, there is fresh paydirt in which to dig.

4. Avoid articles that are intended to inject your own voice into an academic conversation. Write because you have something to say, not because you want to join some conversation among people viewed to be stars, or influential, or powerful in legal academia.

5. Commentaries on social justice, philosophy, society at large, or the social sciences largely are wastes of ink and paper. These surely are important areas, and hotly debated, but see point 1 and also remember that we have Departments of Philosophy, Politics, History, Sociology, Anthropology, Economics, Literature, etc. who focus on such areas exclusively. Odds are that you will not write a better article than such specialists will in the relevant academic field, and an article in a law review article may be interesting to a few people covered in point 4, but really amounts to nothing other than your opinion. (Certain exceptions, of course, apply but these are in the minority.

In short, there could be very good scholarship, but the efforts are misdirected because the authors seem interested in other pursuits.

Jojo

Excellent comment! I would only tweak your last sentence by adding the adjective "legal" before "scholarship."

Note that when professors debate the "value" of scholarship, "worth" in terms of the benefit to the practice of law is rarely mentioned. Professors debate the best way to measure their own beauty, with each preferring the criterion that casts their usually quite ordinary appearance in an unrealistic but flattering light.

@Jojo,

Odds are very, very good that Michael (to take just one example) will indeed write much better scholarship on the history, politics, and probably even economics of patents than will some "specialist" from another part of the academy. A historian can try to tackle patents, but will probably mess up his or her doctrinal analysis. (I've seen it time and time again.) A political scientist can do all the calculations s/he likes on patent law decisions, but that analysis tell us very little without an appreciation/understanding of the nuances of the disputes themselves. Same goes for the economist. Legal scholars *are* specialists… in LAW, which touches on every aspect of life and culture. That this point still has to be explained to anyone in the 21st century is mind-blowing.

Um…

Yes, JD's have the education, training, experience and insight to delve deeply and knowledgably into EVERY ASPECT OF LIFE AND CULTURE.

Who said three years at Yale wouldn't better prepare a self appointed know it all to opine about EVERY ASPECT OF LIFE AND CULTURE???? While Um … trashes the other departments of the University, we all observe that law professors are posing as de facto members of those departments (without, of course, paying their dues). Um…, not unsurprisingly, finds those in his milieu superior to all others.

And, folks wonder why so few read or care about – leaving aside USE IN ANY PRODUCTIVE WAY – the vast majority of the dreck they produce under the guise of "scholarship"? Read the comments above. "I have this many downloads …" "But, I have this many …" "But, I have more over here …" "But, I have a better way of measuring my downloads …"

If only these "specialists" in EVERY ASPECT OF LIFE AND CULTURE would actually pose as law professors and not teenagers seeking attention and validation, and get down to the business, not of self promotion and self regard, of legal scholarship that actually benefits the practice of law.

The underlying issue here is that basically most of the "scholarship" being produced by the legal academy is useless. As "knowledge generation" is the last bastion of defense of the present order, it is telling that better rejoinders can't be found. (The most risible being the argument that such "scholarship" improves teaching.)

Um…

So they *are* "specialists" in general? Amirite?

Doesn't economics touch every aspect of life and culture, at least from the economist's perspective? Same with politics? Same with sociology?

That someone (presumably in the academy) can view the role of a legal academic's 'scholarship' as so clear that it requires no explanation in 'the 21st century' reveals how far we have to go to get those inside the ivory tower to look outside it.

Way to deliberately mischaracterize what I said.

Also, just to clarify, I'm a litigator who spends a whole two hours a week in the "ivory tower" as an adjunct, co-teaching a seminar with a law prof. If that's all it takes for me to have lost perspective, then the academy is indeed far more dangerous than anyone realizes…

Harrison and Mashburn's methods would be desk-rejected by any peer-reviewed journal because they have insufficiently specified their sample selection method. "Two articles" from each of the "top 100" law school's flagship law reviews? What about that doesn't sound like cherry-picking?

Every law professor who purports to do data-driven work needs to read King, Keohane, and Verba, and then probably also Brady and Collier.