Leiter Survey vs. US News

I thought I'd look a little at the differences between Brian Leiter's recent on-line survey of faculty quality and the U.S. News rankings.  First an observation about the changes in the top fifty schools.  There are five schools that appear in Leiter’s 2014 top 50 faculties that are not in the current U.S.News top 50 schools.  They are (and the numbers following the schools are U.S. News rank school and average rank):

University of California, Hastings 55.5 54
Cardozo Law School/Yeshiva Univ 65.5 64
University of San Diego 79.5 79
American University 75 72
Brooklyn Law School 84.5 83

There are also five schools in the current U.S.News top 50 schools that are not in Leiter’s 2014 top 50 faculties.  (And the number following each school is its U.S. News rank):

Wake Forest 33
BYU 37
SMU 42
Maryland 46
Tulane 48
Utah 49

I also looked at the correlations between Leiter's ranks, U.S. News data on peer assessment and overall rank, and Doyle's Washington and Lee Law Review rankings.  The following table reports the correlations between those variables.

  Rank Leiter Peer Rank USN Rank Journals, 2006-2013
Rank Leiter -.94 .84 -.82
Peer 2015 .94 -.86 .82
USN Rank avg .84 -.86 -.67
Journals 06-13 -.82 .82 -.67

For all correlations, N = 49 and p < .0001

I guess one of the things that really stands out for me is the high correlation between Leiter's ranking and the U.S. News peer assessment scores (.94).  As long I as I'm talking about rankings, I want to say that last summer I posted a paper on ranking law schools using entering student data on LSATs, student employment data, and law review citations (which I use as a proxy for academic reputation).  It's available on ssrn.

Update: I've been surprised at the amount of commentary on this post — I guess it shows that people care about rankings, or something.  Anyway, I think people might be interested in the differences between the Leiter rank and the U.S. News peer rank for individual.  Here's a table that lists the schools in order of their Leiter rank and compares them to their US News rank.   A negative difference indicates that the school does better on Leiter than U.S. News; a positive indicates that a school's U.S. News peer assessment rank is better than its Leiter rank.   I noticed that my own school (UNC) does substantially less well on the Leiter rank than on the U.S. News peer assessment rank.  Actually, I think UNC has the fourth-largest diffence between Leiter rank and U.S. News peer assessment rank (after Washington and Lee, which does much better on U.S. News than Leiter, the University of Illinois, which does much better on Leiter than U.S. News, and the University of Florida, which does much worse on Leiter than on U.S. News).  I guess I should have voted.  What's the slogan?  "If you don't vote you don't count," or some such.

At the risk of sounding too self-promotional, this paper uses citations as a replacement for U.S. News peer assessment, which — as some people point out in the comments — rely on raters to have some knowledge about a bunch of schools.

39 Comments

  1. Jojo

    Who really cares about Brian Leiter's rankings? Or anyone's rankings for that matter?

    Academia is a strange, prestige-obsessed universe.

  2. Anon

    Let's be clear about what is being ranked. This is really perceived scholarly distinction (it's actually even narrower – perceived scholarly distinction based on a particular style used to approach an issue). The fact that "faculty quality" is used to describe perceived scholarly distinction is unfortunate. Prospective law students should be more interested in teaching ability, a school's teaching philosophy, output measurements, and curricular options. Jojo asks why anyone cares about rankings. Well, because prospective law students select schools based on the rankings. An absurdly high percentage of the U.S. News rankings boils down to scholarship – most use it to assess their peers' "reputation" even though they know very little about their peer institutions. If that were to change, faculties would feel pressure to adjust their time allocations.

  3. Amir

    Prof Brophy, why did you ignore UC Irvine in your discussion? It is in the top 20 in Leiter's poll but unranked in US News. Seems like it should be mentioned, as it does affect your thesis of high correlation.

  4. Steven Freedman

    I'm perplexed at Princeton Law's absence from this list, they usually finish top ten 🙂

    Seriously though, I think we all wonder how someone could possibly have a sense of the difference in scholarship quality among such a diverse group of schools. I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane. Or Emory over Wisconsin. That doesn't mean the survey's not a fun exercise, just that it's value is dubious.

  5. anon

    Steven

    You're right. Many of the persons voted for their alma mater, or for other frivolous reasons.

    The persons who "voted" don't know much of anything about the "scholarship" at all these schools. At most, they may have some real knowledge about a few persons writing in "their field," and beyond that, some scant, cursory knowledge of the work of one or two professors at a few of the schools. Nothing upon which anyone would claim an informed judgment. This is like grading papers based on the names of students whom you have never met, or worse (and this is par for the course in legal academia) students about whom one "heard" something.

    People produce these beauty contest-like rankings because it garners attention. The post preceding the one considered here (on that other blog) dealt with an attack on a piece by Paul Campos that Campos wrote six years ago(!), with no apparent reason to have pulled it out of the dust heap other than to brandish the headline "Paul Campos admits he is fraud."

    Now, there is some news you can use.

  6. JPQ

    Steven makes a good point that this is a ranking by people who read Leiter's blog, and therefore, for example, have been exposed to that blog's cheerleading for the recently-established UC Irvine law school.

  7. ATLprof

    "Academia is a strange, prestige-obsessed universe."

    Yes, it is.

    But let's not hide the fact that academia is far from alone with respect to rankings. Prospective, current and former students are just as obsessed if not more obsessed with law school rankings. To a lesser but still significant extent, employers are as well. It's a huge problem.

  8. Alfred Brophy

    Hi Amir,

    The reason I didn't mention Irvine is because it's a relatively new school, which has neither a USN rank nor sufficient data for its law review.

  9. anon

    Students follow; they do not lead.

    Rankings aren't used by "employers"; like all academics, the generalizations about the legal profession based on a small sliver of it are so tiresome.

    Some "employers" look for the "best and brightest" and use the proxy of admission to a "top" law school. But this has to do with the selectivity of the school's admission process; this has nothing to do with "rank." Nobody cares if it is HYS or SYH or YSH or whatever.

    Moreover, the law profs who pontificate about "opportunity" law schools contradict themselves regularly by claiming that employers follow "ratings" … as these schools are largely unrated, i.e., not even recognized sufficiently to rank.

    Again, "rankings" are irrelevant. Employers look to the selection process. If a school is admitting persons who can't pass the relevant tests, employers will take note. Students want to attend "top" law schools, to be sure; but this has more to do with endowments and longevity, as many studies have pointed out, the pecking order was established long before USNWR.

    "Rankings" are an obsession based on caprice. As stated above, the ranking mentioned in this post was based on basically NOTHING reliable. As such it is a useless activity engaged in by folks who have nothing better to do.

  10. Orin Kerr

    Steven Freedman writes: "I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane"

    San Diego has a marvelous faculty with a large number of active an excellent scholars. I've presented papers there before, and I received some of the best comments I have ever received. Tulane is a good school with some fine scholars, to be sure, but I don't think it has nearly the quality of the faculty at San Diego.

  11. anon

    Orin

    You received "some of the best comments" about a "paper" that "you ever received"…? And, that is supposed to be accepted by any serious person as a basis for ranking a faculty? Really?

    You couldn't make this stuff up. The total nonsense that actually passes as knowledge in legal academia is so consistently risible. That any thinking person would accept the statement quoted above as a basis for ANYTHING, leaving aside "ranking" a law faculty- and I have no doubt that just such nonsense played a big role in the "ranking" under discussion – is a sad testament to the juvenile nature of legal academia.

  12. Anon too

    Anon, surely you know that legal scholarship is based on anecdotes. The "See e.g." cite is the legal scholar's version of a correlation coefficient, a high R-squared, or a t-test. By the way, I have eaten at the McDonald's in San Diego and it was quite good. I have not eaten at the one in New Orleans and I am sure it has some fine burgers but there is no way it could measure up to the quality of the ones in San Diego. Anon, that is what we call "law science."

  13. The One and Only Anon

    The negative comments here about San Diego show the continuing problem about this site and the debate overall – it is filled with people who are know nothings. Serious faculty have long respected USD as a significant and creative place as Orin makes clear. The school has the good fortune to be located in one of the most beautiful places imaginable for a school but in a market that is quite small (altho the biotech boom there has not hurt). They overcame the structural disadvantages through years of hard work led by a very effective dean.

  14. anon

    "Serious faculty have long respected USD as a significant and creative place as Orin makes clear. The school has the good fortune to be located in one of the most beautiful places imaginable for a school but in a market that is quite small …"

    Yes, indeed. If the readers are not yet howling with laughter, then they aren't seeing the supreme hilarity of these comments. Unbelievable. Just unbelievable.

    Let the rankings begin.

    "Let's see: San Diego, so sunny, but sort of small, but there is a "biotech boom" there … (same person who last week was saying TJLS's building isn't worth anything) Hmmm. I'd rank that faculty at 61.2, with a margin of error of .2, but only because I'm not sure about the whole burgers thing. Burgers can be quite tasty, as I'm sure that Orin would agree. Orin received some great comments there. I heard him say so. Make that 61.4."

  15. MacK

    I'm not sure if this will be filtered out since I seem to again be PNG around here – but a few months ago the THE™ World University Rankings 2014-2015 and the QS World University Rankings® came out and in a number of countries were frontage news – with either considerable vaporings about how universities had slid or self congratulation about how universities had soared in the rankings. Since these rankings are begging to have a real world consequence – for example impacting the allocation of ESF funding and perhaps NSF, I became curious as to the methodology. Oddly enough, intellectual property law and antitrust/competition law uses surveys so you learn how to spot bad methodologies.

    To say that the reputation surveys are it is as close to Horseshit Statistics™ as can be imagined would be kind. Let me explain – both surveys operate in similar ways – and a key aspect is the peer survey. In this they send thousands of academics a list of universities and ask that the academics rank their top 10 or 15. They ten rank universities by the number of times they appear in the top 10-15 selected. Not surprisingly, it seems that the survey participants tend to rank the schools that they have heard of – and since every academic has heard of Harvard, Oxford and Cambridge – well they get on about 100% of the replies – then everyone ranks the university next door and maybe the last place they went to for a conference.

    Universities are aware that these surveys matter – a lot – in terms of grant revenue, the ability to attract international students (who in Europe and other countries where tuition is heavily state funded are very valuable because their tuition is essentially unregulated – they can be charged what the market will bear (close to full tuition at a US college), rather than what the department of education allows.) The result is that, at least within the top 100-200 gaming of the rankings is becoming visible. In the case of the QS ranking it seems that getting just 2-3 more people to list a university in their 15 will rocket it up the rankings (since the actual counts for the schools below say Harvard, Cambridge and Oxford are in the single digits.) This year Australian universities seem to have soared in the rankings – for no clear reason other than one suspects good marketing.

    I tried an experiment on a few physicists I know – asking them to rank the top 15 physics departments – after MIT, Cambridge and CalTech almost all were places that the physicist worked with professionally – they were just the departments that they could remember. How many readers on this forum had even heard of Florida Coastal or Ave Maria (unless they applied for a job there?) How many people read most law review articles … and how many who are not professors writing on the same subject (or the author's mother?)

    The hard reality is that these surveys would be pointless – if they did not attract readers the way top ten lists do in Vogue and Cosmopolitan (not to mention the Weekly World News and other checkout fodder.) Pity that people base important decisions on them.

  16. James Grimmelmann

    No one is an expert on dozens of law schools. But even if professors have a lot of information only on some schools, they have some information on a lot of schools. They know the faculty in their field at most schools; they read the occasional paper from scholars outside their field; they give workshops and get comments, as Orin did. The point of a survey is that it aggregates the information held by many people to give an overall picture. When dozens of respondents in different fields at different schools all rate the University of Mars over Trantor Tech, it is fair to say that U of M has a better academic reputation than TT, even when none of the respondents has a complete basis for the head-to-head comparison. It is not necessary that every respondent know everything; only that most respondents know something.

    That's not an endorsement of the U.S. News or Leiter surveys, both of which have issues that have been discussed to death. It's just to say that professors can easonably express an opinion based on their individual experiences. Enough anecdotes are data. All knowledge starts somewhere.

  17. Camilla Highwater

    Three respondents to Leiter's survey rated USD's faculty as superior to Yale's. So it must be really good.

  18. anon

    And, all of those votes came from profs at USD, no doubt, who chose their employer over their alma mater, unlike most.

  19. anon

    Yes, all knowledge about the "strength of law faculties" starts with anecdotes and rumors, but, more importantly, finds the true bases of supposedly valid "rankings," at the top, on a long standing pecking order (that preceded USNWR and perpetuates itself by means of faculty hiring, endowments, longevity, etc.) and at the bottom on absolutely nothing at all.

    The unbelievable group think in which law academy engages is something to behold, it truly is. Never has there been a group of otherwise mainly intelligent people who so often appear to be willfully ignorant and intellectually self-indulgent.

    "Mirror mirror on the wall … Who is the fairest of them all?"

    "Why, it is me! ME! ME!"

    (Well, "me" when compared to Tulane; and this, I know for sure. Orin Kerr said he received great comments from my colleagues, and that means everything if we add up a bunch of Orins!)

  20. Paul Horwitz

    Anon, is your complaint about rankings, or is it about stating any opinion about the scholarly quality of a law school's faculty based on a scholarly interaction with the faculty of that law school? I certainly understand the former but I guess I find the latter a little odd–and, given that Orin was just politely answering someone's question in the comments section of a web site, and not doing something of lasting or major significance, the quantity and heat of your multiple responses also seems pretty remarkable.

  21. anon

    Paul

    As I said above, agreeing with Steve Freedman (at least in part):

    "You're right. Many of the persons voted for their alma mater, or for other frivolous reasons.

    The persons who "voted" don't know much of anything about the "scholarship" at all these schools. At most, they may have some real knowledge about a few persons writing in "their field," and beyond that, some scant, cursory knowledge of the work of one or two professors at a few of the schools. Nothing upon which anyone would claim an informed judgment. This is like grading papers based on the names of students whom you have never met, or worse (and this is par for the course in legal academia) students about whom one "heard" something."

    Thereafter, profs chimed in to validate every one of these observations, in a rather embarrassing way.

    So, to answer your questions:

    "Anon, is your complaint about rankings[?]"

    Yes. As stated above: "supposedly valid "rankings," at the top, [are based on] a long standing pecking order (that preceded USNWR and perpetuates itself by means of faculty hiring, endowments, longevity, etc.) and at the bottom on absolutely nothing at all."

    "Anon … is it about stating any opinion about the scholarly quality of a law school's faculty based on a scholarly interaction with the faculty of that law school"

    No. Not any opinion. That seems to be a bit of an overstatement, Paul. I signaled Orin's anecdote about "great comments" as a perfectly representative bit of nonsense that is perpetuated as reliable "fact" in basing a judgment about the "strength of a faculty."

    You object to "heat." I'll let that statement stand, rather than going thru these blogs and quoting some of your comments.

  22. The last Anon

    I'll also put the "heat" matter aside but to the extent Anon is bringing the heat, I feel like many are striking out. The idea that knowledge starts with a set of anecdotes is silly. That's not knowledge unless the knowledge is being able to repeat the anecdotes. One person may say, I got good comments at Georgia State and from that I infer it is better than Mercer. Another may say. I got better comments from Mercer than Charlotte. And then someone says I got better comments at Charlotte than Georgia state. Those anecdotes could all be true but that does that mean we know anything about the quality of law schools. Plus there is the voting for your school or the one from which you graduated issue. Wasn't there a ranking a couple years ago in which most FSU profs rated FSU above all other law schools? Leiter's ranking is simply a report of what people said when asked. Nothing more.

  23. Paul Horwitz

    Thank you for your response. Just to be clear, I have no objection to criticisms of various forms of ranking, or rankings altogether, for various reasons including the ones you offer here. In particular I agree with some of the reasons you offered, but even if I didn't agree with them, I think this is a perfectly legitimate subject of criticism. And although I do believe commentary should generally be civil, and anonymous commentary in particular should be civil, I don't object to heated replies as some kind of absolute matter, although sometimes I think a fairly innocuous comment ends up getting criticized in a surprisingly and unduly heated manner for various reasons (because the commenter is passionate about that issue in general, for instance).

    In this instance, and I am happy to leave it at a polite disagreement with you, I thought Orin's comment was fairly innocuous and, whether you think it representative of bad answers faculty have given in other cases, can't be so described in this instance. In this particular instance, I didn't take Orin to be endorsing rankings in general, endorsing his experience with San Diego as a basis to fill out some actual ballot somewhere, offering advice on which of these two law schools to attend, or anything as significant as that. I took him to be answering a specific question that someone actually posed in seeming hope of receiving an answer: given a particular survey about the quality of faculty, and setting aside the larger questions whether that kind of thing is worth measuring or capable of measure, why might someone rank San Diego higher than Tulane? (Of course, if that's not what Steven Freedman meant, or if he personally read Orin's answer differently than I did or was offended by it, he's welcome to weigh in. He was the person the answer was intended for, after all.) I take it that Orin's answer, which was based on personal interaction with that faculty in the area–scholarly quality–that the questioner was asking about, wasn't meant to do more than offer one answer to that question. It may have been quick and dirty, but I didn't find it particularly overflowing with self-regard or assurances as to absolute reliability. As long as that's all it is, I don't find it terribly outrageous that someone offers such a question based on personal experience. I was thus surprised at what I thought was the vehemence and sarcasm ("me, me, me" and so on) of your answer to the extent it was directed at Orin. I tend to think it's nice, and a pleasant contrast to many online discussions, when people use comments to ask particular questions and others actually offer particular answers. If you felt otherwise about his answer, so be it, I guess.

  24. The Pentultimate Anon

    This is very interesting. Orin has treated the matter exactly as it should be treated. He had his say; Anon has had his say and neither needs a defense attorney because neither probably cares. But since Paul seems offended on Orin's behalf (or is something else afoot) let's get the facts straight:

    Q: Seriously though, I think we all wonder how someone could possibly have a sense of the difference in scholarship quality among such a diverse group of schools. I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane.

    A:San Diego has a marvelous faculty with a large number of active an excellent scholars. I've presented papers there before, and I received some of the best comments I have ever received. Tulane is a good school with some fine scholars, to be sure, but I don't think it has nearly the quality of the faculty at San Diego.

    So Steve's question is "what factors." Orin's response is a non sequiter. A factor cannot be that he got good comments from one school while at the same time having no responses from the school he ranks second. And then on Tulane; "I don't think is has nearly the quality of the faculty at San Diego." Unless 1)Orin has had comments from every school and 2) comments are the principal measure for ranking, it's not responsive to the question. That leaves us with he does not "think" it has nearly the quality. When Orin gets Lieter's ranking he puts San Diego over Tulane on the basis of comments and a hunch.

    I do not what to put words in Anon's comments but I think his beef is that no one answered the original question and the purported answer is irrelevant and hunch.It's like the last Sienfeld episode — the rankings and this discussion are about nothing but the deep involvement of those who are law professors, who in a bind play the civility card as Paul does, does tell us volumes.

  25. anon

    Needless to say, I agree with TPA.

    And, to respond directly to Paul's comment: "I didn't take Orin to be endorsing his experience with San Diego as a basis to fill out some actual ballot somewhere … or anything as significant as that."

    I did.

    And Paul, to bring a little "heat" here, you understood Orin's response the same way, as you say, "I took him to be answering a specific question that someone actually posed in seeming hope of receiving an answer." That question, as TPA points out, was: "I'd be thrilled to hear someone articulate exactly which factors went into ranking San Diego over Tulane." You are dressing up your comments quite a bit here, but the lipstick doesn't really cover the sort of risible nature of the Orin answer.

    Is it significant that it was Orin? No. His comment, as repeatedly stated above, has been singled out because it was, as you say, "representative of bad answers faculty have given in other cases." My only disagreement here would be to substitute the word "risible" or "ridiculous" or perhaps even "self referentially offensive" … Not civil?

    I find uncivil arrogance and pomposity. Law profs have a bit too much of it, and too little humility. I believe this hubris is destroying the legal academy. There is little that can be done, but calling it out is a privilege so far afforded here, at least for now.

    Of course, the response of any person whose words are blown up is "it was just a little, off hand remark." Think of how many times, however, you have rejected that excuse, society has rejected that excuse. Is Orin personally at issue here? Again, no. But let's not pretend that his comment wasn't noteworthy either.

  26. Orin Kerr

    Sorry I was away from the thread for a few days; kind of hilarious that my comment drew such a reaction. To be clear, I didn't base my view of San Diego just from presenting a paper there. I based it on following the scholarly work of the San Diego faculty for about 15 years, and also presenting a few papers there. San Diego has a lot of very productive top scholars, and I read their work often; I don't think it's nuts to have an opinion of the work I have personally read over a 15 year period supplemented by the experience of engaging with the faculty in a workshop a few times.

    Anyway, sorry I didn't give all of the details of why I think SD has such a good faculty in my comment. It didn't occur to me that anyone cared, or was going to make judgments about the lack of details in the comment. Lesson learned, um, I guess.

  27. Orin Kerr

    Two more thoughts, in case anyone is still reading:

    1) I assume it's obvious that when a person asks to compare two things and say which is better, they are only giving a personal opinion based on their own knowledge and experience. If you ask people to rank the best restaurants in town, or the top college football programs, or the best candidate running for a particular office, the answers they give will always be partial and quirky. I take it that's why such rankings usually ask lots of people and average out the answers or do some assessment of the collective view. So I don't think that expressing a view of whether one thing is better than another based on one's own partial and quirky experience is arrogant.

    2) FWIW, I didn't complete the Leiter survey because I don't personally have views of the faculty at most of the schools. I happen to have knowledge of a few schools, and I think I can make broad comparisons between them. But I certainly don't know enough to rank all of them, so I didn't complete the Leiter survey.

  28. Paul Horwitz

    I must admit that I find incivility uncivil, and arrogance and pomposity arrogant and pompous. However I feel about either quality, I've never felt much of a need to confuse them. And, what with my trying to coexist with other people, who are human, I try to assume that some potentially objectionable remark was innocuous and offhand as often as is reasonably possible, which is an awful lot. Don't you? Otherwise I would go around getting offended all the time, for no especially good reason.

    Penultimate, thank you for your comment. Nothing else was afoot other than being saddened and unsettled by what I saw as the commenter's high-dudgeon overreaction. I do appreciate your points and will think about whether I misread the context of the question and answer. I will add three things, though. 1) While I take your point about no one needing a defense attorney, that's not how I saw what I was doing. One might well conclude that I was butting in; if that's so, though, I wasn't alone in all this, since the anonymous person raising the fuss was not the person asking the question that Orin answered in the first place. If we're thinking in terms of people having their say, weren't those the only two relevant people? Wasn't everyone else beside Orin and Stevens just a buttinsky? 2) If you're right in your description, it seems to me a commenter could have written, "Orin, I don't think you answered the question." Wouldn't that have sufficed? 3) I have no problem with criticizing the rankings. But, as I wrote above, I didn't take Orin answering a commenter's question in an offhand fashion as an indication that he had filled out a ranking survey at all, let alone done so on that basis. I kept thinking I'd missed an earlier remark somewhere where he'd written, "Hell yes, I filled out the survey." But, as he confirms above, he didn't. He really was just answering, of failing to answer if you read it that way, an individual question on the comments site of a web site.

  29. anon

    Orin

    You still don't seem to get it, so your attempt to minimize the way you responded to Steve's question is not convincing.

    You stated, in response to a question by Steve asking for a valid list of factors that would justify a reason to rank USD over the faculty at Tulane, as follows:

    " I've presented papers there before, and I received some of the best comments I have ever received."

    Praise is owed for working the word "I" into such a short sentence three times. But, was the fact that YOU received “great comments” when YOU presented a paper there a great response to Steve's question?

    Now, you are doubling down. We are supposed to believe that you have some way to compare the entire faculty at USD with the faculty at Tulane because YOU presented more than one paper there! Well, there you are, Orin. Case closed.

    Really, Orin, you go on to assert that ranking top college football programs is "always partial and quirky." Again, you just don't seem to get it: ranking football teams is usually the result of, well, let's think a long time about that. Steve was looking for criteria. You were thinking of the fact that you received great comments at USD.

    Now (while protesting that you don’t think anyone is reading comments, etc.) you've added: “I've read papers by faculty as USD." Yes, and so? Is that the basis for saying "Tulane is a good school with some fine scholars, to be sure, but I don't think it has nearly the quality of the faculty at San Diego."

    What was the basis for that sort of, to use Paul's favorite phrase, uncivil and completely unsupported remark? You have offered nothing worth considering in response to Steve question: only a sort of arrogant assertion backed up by a slam on one faculty and bogus reasons for concluding another is "better quality" (what a joke this all is!).

    As several comments above demonstrate, these surveys are nonsense, and aggregating opinions of the type you describe does not validate such surveys, it destroys any putative value associated with them. The fact that you believe that aggregating garbage is valid is so disappointing.

    I think most reasonable people would agree, perhaps even you, that the value of the survey under discussion is basically zero. If you really want to pursue the arguments you are making here to support it, then I would say you are casting your lot with it, and the worth of your say so on these matters should be judged right along with the worth of that survey.

    The move here, Orin, is not to pretend you just weren't thinking, but to reconsider your stance. YOu are correct, your comment standing alone doesn't mean too much to anyone.

    But, your comments reflect an overall group think that should be examined and, hopefully, put in a context that shows surveys of this type to be too flawed to serve any purpose other than the one to which you put this one.

  30. Orin Kerr

    anon, one of us is seriously misreading the other, so let me approach this a different way with this question: If someone were to ask you what is the best restaurant in the town where you live, how would you answer, and based on what?

  31. Former Editor

    anon,

    Is your objection to ranking at all or to the particular methodology used in this particular survey conducted by Leiter? If it's the former, I'll suggest that you leave those windmills alone. Some kind of ranking is going to exist whether we like it or not. If the latter, do you have a constructive suggestion? i.e. what criteria do you think would be relevant to a ranking of faculty and how would you propose measuring it? There are a lot of ways of "ranking" legal scholars/scholarship. Leiter's own website has used at least four I can think of off the top of my head, most if not all of which have some serious problems.

  32. Steven Freedman

    Whew. That's a lot of back and forth for my reasonable question and Orin Kerr's perfectly reasonable response.

    Truth be told, I don't have any novel thoughts on this issue as I think most people are on the same page here. In a nutshell, I think most people agree that rankings are important to the extent that prospective students, employers, etc. often use them as a tool for comparing schools. Some of them are well constructed enough that they may even have validity to them. At the same time, I think most people, even those relying on the rankings, understand that many (most?) rankings include methods and data best compared to poppycock.

    Don't get me wrong. I like Leiter's survey. It reminds me of those college bowl public radio fund drives where you root for your school to beat out your rival. It's the only fund drive I like. But the survey is silly. Other than making it a full-time effort, there is no way for a typical faculty member to keep track of the scholarship quality for 83 separate law schools. It's a popularity and anecdote contest, not a realistic evaluation of faculty quality. I'm not even sure Brian Leiter is suggesting otherwise.

  33. The Singing Fatman

    "Nothing else was afoot other than being saddened and unsettled by what I saw as the commenter's high-dudgeon overreaction."

    Yes, I had to get counseling myself when I read Anon's comment. Fortunately, Orin, rightfully so, does not give a you know what and I think Steven has it right. The Leiter thing may be more of a reflection of those filling it out while knowing they know so little than what they might make of it after filling it out.

  34. anon

    Orin seems to think that ranking the strength of law school faculties is just like ranking taste in food.

    Nuff said. Case closed. Just like Orin's mind.

    Oh, and BTW, Orin, for a bit of humor here, watch the criteria used by Gordon Ramsey in his British show to rank the "best restaurants": service (using hidden cameras and tested by experts posing as difficult customers), ability to serve 30 people in two hours start to finish, a fly speck inspection of the kitchen for cleanliness and design, a competition to prepare a meal for the best known critics and consumers in each category, etc., etc., etc.

    Funny, nowhere did Ramsey mention or use as a criterion whether anyone made "great comments" about his restaurants. Funny that … as big an egomaniac as he is, even he knows how ridiculous that would have sounded as a criterion.

    As for the "survey" I think there is a consensus that it is worthless. Orin, you apparently will defend it to the last, and others will defend you. Perhaps then, in the final analysis, you personally win the "FL survey" but, of course, lose on the merits. How fitting.

  35. The Singing Fatter Man

    Anon, you are making so many of us so very sad. Please stop.

  36. Barry

    In addition, Orin, for prospective students, there are two criteria which matter:

    1) Job prospects.
    2) Cost of attendance.
    (note, these can be modified for regional desires).

    Everything else is bullsh*t for a student, whose life can be destroyed by wrong choices.

  37. Orin Kerr

    Barry, the Leiter survey is just a faculty ranking of the faculty at law schools. It does not ask nor answer how students should decide where to enroll if they are accepted.

Leave a Reply

Your email address will not be published. Required fields are marked *