Genius, Defined; or, What Is a Law Degree? (Part II)

This continues our exploration of Simkovic & McIntyre's Economic Value, a masterpiece of monumental thought leadership from two of the American northeast's leading educational institutions.  If only Richard Epstein and Laurence Tribe taught in New Jersey, we'd have so much social justice and constitutional understanding, the justice system would literally grind to a halt to marvel, like Narcissus, at its own beauty.

"Genius" is a subjective term, absent some quantitative measurement like an intelligence quotient test that would omit at least half of what we consider genius and include some on-paper-geniuses that, in the common sense, do not qualify.  As a result, studying "genius" in a population is quite challenging for researchers.

"Law degree," however, would seem like an objective term: a juris doctor degree from an ABA-accredited school.

This definition is essential to a study like Simkovic & McIntyre's.  Not only does it ensure that a properly-configured population is being studied, it also allows for subsequent researchers to repeat Simkovic & McIntyre's observations.  If scientist A discovers something amazing, he needs to leave a blueprint so that scientist B can follow.

However, in attempting to replicate and confirm Simkovic & McIntyre's Earth-shattering conclusions, one (read: the LSTC) runs into an immediate problem:  What the heck do they mean by "law degree?"

This isn't a trick question or quixotic philosophical detour.  The term is in the title, the abstract uses the phrase eight (8) times, and it's the main variable being studied.  In common reporting on the study, the term "law degree" appears to be assumed synonymous with the juris doctor.  (See, e.g., the ABA Journal write-up).

But here's the rub:  Simkovic & McIntyre do not appear to have measured juris doctor holders.

The problem here is that the Census Bureau's Survey of Income and Program Participation ("SIPP"), used by Simkovic & McIntyre to build their sample set, does not appear to ask respondents whether they have a juris doctor.  Instead, one of the SIPP topical modules (Wave 2, as viewed with the Census Bureau's DataFerret tool) asks respondents whether they have a "professional degree."

"Professional degree" does include the J.D as one example of a "professional degree," but there are two other categories that law graduates may possibly select outside of normal response error:  master's and doctorate.  What Simkovic & McIntyre appear to have done is cross-reference that answer with a separate question on the topical module asking what "field" the individual received that degree in.

In a footnote (fn 16 in this version, which may be a draft?) (see below), Simkovic & McIntyre explain that they required respondents to identify as having a "professional degree" in the field of "law."  They elaborated that they excluded some unexplained number of people identifying as having a master's or doctorate or "imputed to be lawyers."

This approach may be both over-inclusive and under-inclusive in context.  It's over-inclusive because there are other "professional" degrees in the field of law besides the juris doctor.  Foreign-educated attorneys, for example, may rightly claim to have such degrees.  Additionally, it is impossible to know if the sample is polluted with advanced paralegal degrees or others who have any of the myriad degrees that have spawned in the last twenty years.

It's under-inclusive because it's incredibly likely that some juris doctor holders checked a different box.  Don't take my word for it; look at the actual numbers.

Looking at the 2008 SIPP responses and comparing people who selected "law" as their field of study in the Wave 2/topical module among various types of degrees identified, one sees the following:

Master's:  54
Professional School:  371
Doctorate:  67

By their explicit words, Simkovic & McIntyre apparently only included the 371 "professional" degree holders.

But what about the 67 people who identified having a "doctorate" in "law"?  Are we assuming that none of these people have a juris doctor?  Because I would imagine almost all of them do - it isn't like traditional PhD's in "law" are common (particularly because "social sciences" only has 33 and "liberal arts" only has 14 for the 2008 sample).

From a statistical purity standpoint, we have a big problem: people who almost certainly should be included are purposefully excluded by the study's apparent design.  Not just 1 or 2%, but it's quite possible that 15% of the 2008 SIPP juris doctor holders were excluded as not having "law degrees."

And that's why the definition matters.  When we ask law schools to collect data on their own graduates, we know those individuals have juris doctor degrees.  If we exclude someone, it is because (in theory) we lack the data.

When Simkovic & McIntyre infer that a person has a "law degree" based on having a "professional" degree in "law" in census records, they introduce two layers of response error and use imprecise terminology.

The result is that the Economic Value may include people who don't have a juris doctor, almost certainly excludes people who do have a juris doctor, and tries to implicitly slide by on the premise that what they infer to be "law degrees" based on an unrelated governmental survey actually are people with juris doctor degrees.

Simkovic & McIntyre have repeatedly emphasized that other studies are over- or underinclusive and theirs hits the sweet spot:

Our data sources enable us to estimate earnings premiums and increased labor force participation attributable to a law degree, not only for the under-inclusive category of lawyers or the over-inclusive category of professional degree   holders,   but   for the   appropriate group, law degree holders.

Economic Value, SSRN Version, p. 12.

When you consider that the tracking of lawyers includes only people who have a JD or the equivalent, or that standardized law school tracking already includes the exact, dead-on population, such a claim appears to be totally unsupportable bullshit in light of SIPP's limitations of definition.

That Simkovic & McIntyre - and numerous others - have persisted in using this study to purportedly support conclusions about the population of ABA-accredited juris doctor holders - even though the study's explanations of how it defined such a population appear to be unacceptably imprecise on their face - is a testament to Simkovic & McIntyre's slick genius.

Most scientists would never be able to pull off using a bad, imprecise data set as a superior means of study than using records sampled from the exact population.  The most renowned scholars on Earth - Hawking, Dawkins, Andrew Fucking Wiles - would never dare pass off an imprecise definition as the one true authentic representation of a million-person population.

But Simkovic & McIntyre did.  It may significantly alter the results of their (still-otherwise flawed) study; it may not have any effect at all.  The fact that it exists without any real explanation shows that Simkovic & McIntyre clearly belong in the "genius" category regardless of how we define the term.

 Note:  there appear to be multiple versions of Economic Value circulating on the internet.  If anyone is aware of a "final" version that addresses any issue raised in this post or any subsequent post, please comment or forward a link to my attention.  In any event, the author expressly disclaims any purported factual allegations or implications in this or any other part of the series and states affirmatively that he/it is a moron of improper breeding and education and that readers should defer to the educated professorate to guide our thoughts.