Genius Uncontrolled, or, The Control Group Problem (Part IV)

This continues our series on Simkovic & McIntyre's still-cited masterpiece Economic Value as part of a pre-set July publication schedule.  For the teeming tens of thousands of regular readers, a reminder that new content (or should I say "content") will return in August.

In Economic Value, Simkovic & McIntyre stress that they are looking for a premium provided by the "law degree" rather than an absolute return on investment or absolute earnings.  That's a fine theoretical approach ("much like law school, it works on paper!"), except there are a number of practical problems with their chosen methods.

First, a "law degree" costs time and money.  Lots of it.  Simkovic & McIntyre address the return on investment part in Economic Value, but they seem to understate the costs associated with acquiring the premium.  For the "best and brightest," three years of a working life at the start of a career will easily eat $120,000 or more of nominal earnings that can be invested for longer terms than money made later in life.

Second, they chose a terrible baseline.  Simkovic & McIntyre gauge the premium of the "law degree" by comparisons between the "law degree" and what they call the "terminal bachelor's holder."

What they are trying to do is use a "control group" to evaluate the effect of a certain variable (in this case "law degree") within a broader population.  For the lawyer who hates mathematics and science like an uninsured tortfeaser, a control group is basically an attempt to find something that is comparable and otherwise isolates the variable seeking to be studied to rule out any other explanations in advance.  Assume you want to study the effectiveness of a particular brand of pesticide on apple trees.  To do that, you would need a group of apple trees that do not have the pesticide, but otherwise match up well in other variables (like age of tree, climate, rainfall, soil chemistry, insects and other predators present, etc.).  The more all other possible variables match up to a group that does have the pesticide, the more one can conclude that any differences between the two studied groups (the control group and the experimental group) are due to the variable.

By analogy, if we're studying the positive effect of a "law degree," we need to find a group of students who are otherwise identical to the group of people who landed "law degrees," but chose other paths instead.

The comparison between terminal bachelor's holders as a control group and "law degree" holders as the experimental group is therefore bogus (obviously, and, frankly, making such a comparison appears to lack complete intellectual honesty) for several reasons that immediately come to mind.

•  A "law degree" isn't just a extra sheet of paper.  It costs three years to acquire with focused study in a particular field.  Thus, the proper control group would be bachelor's degree plus three years.  It isn't like "law degree" holders can be compared with 22-year olds.  The oranges-to-oranges comparison would be bachelor's degree holders who are twenty-five or older - people who could actually have a "law degree" from a raw time-elapsed perspective.  Indeed, in Economic Value, it appears Simkovic & McIntyre included all bachelor's degree holders as-of the start date of the survey, but excluded those on their way to getting a law degree (as naturally someone in law school would not report having completed a professional degree in law).  That means people who are 23 and for whom it would be mathematically impossible to have a law degree are in the control group.  That's bad.
• People who are eligible for a "law degree" have to be admitted to law school first.  The requirements for law school (much less earning a JD) are more stringent than having a terminal bachelor's degree, and they likely always will be even as applicant standards plummet.  So why are the two being compared?  Simkovic & McIntyre hint at this with their discussion on ability sorting, but ultimately fail to use any reasonable method of using admissions screening to build a control group.  Traditionally, law schools have only picked from the top third or top half of bachelor's degree holders.  The proper control group here would be bachelor's degree holders who could have reasonably been admitted to law school.  A "law degree" holder who graduated in 2006 has no business being compared to a 2003 college graduate who had a 2.3 GPA from Backwater State and a 134 on the LSAT.  Yet, Simkovic & McIntyre apparently have no compunction making the comparison.
• People who don't go to law school can often get other forms of advanced study.  A JD takes longer than an MBA or numerous other masters degrees that could boost earnings just as much, if not more.  Limiting the "control group" to folks who terminated with a bachelor's degree is arguably methodologically dishonest, as it excludes arguably the most comparable educational options - and one that a lot of JDs would actually have done had they not gone to law school.  I understand excluding medical and dental students, but a proper control group shouldn't be limited to just terminal degree holders.  SIPP identifies master's holders in non-law areas.  Why on Earth are these excluded from the control group?  If anything, MBA students would provide a significantly better baseline than terminal bachelor's holders.

Thus, the bogus control group - i.e., including people who would be ineligible for a "law degree" for multiple reasons and excluding people who most share characteristics with the studies group - is yet another reason Simkovic & McIntyre's Economic Value research can be (and should have been) safely disregarded as bullshit, cutting-room floor material not even worthy of the special director's cut DVD.

At this point, we're only on Part IV of our series and we've identified a definitional problem with the population/sub-population being studied, a sample size problem for the sub-populations divided by yearly cohorts, and a problem with a control group that makes no intuitive sense and leaves open a variety of variables that may be affecting the study results (for example, that the smartest terminal BA holders do just as well as those with law degrees?).

I think that what Simkovic & McIntyre did in this case is remarkably courageous and stupendously brilliant.  Knowing that most lawyers are stupid when it comes to math and science, and knowing also that a law degree makes quite a bit of money over the long haul, Simkovic & McIntyre found an insightful and ingenious way to shit on paper and prove statistically that a law degree was a move of champions.  The ends justify the means, after all.  Cavalier statistical mis-steps be damned - this paper does a public service.

Again, that is the hallmark of true genius.  Someone should cut these gentlemen a MacArthur grant yesterday.