This concludes our month-long review of Simkovic & McIntyre's Economic Value. This blog series added some new things I did not see previously observed and rehashed some deserved "criticisms" of the work by those who fail to appreciate its true greatness.
For this last entry, I want to praise Simkovic & McIntyre for their greatest feat of all: perpetuating the great lie.
As I sit here, drafting this entry, I note that my present lawyer salary is about two times greater than my pre-law school salary according to my last full year's tax return before enrolling in an institution of third-tier excellence. On purely economic terms, presuming I make this salary the rest of my natural life, law school was a positive economic investment.
And yet, my professional career has never felt this dead, and I would kill a roomful of adorably fawning puppies to go back to age 24 and try it all again.
I don't hate practicing law. I love writing, researching, thinking through problems, and working with about two-thirds of the people I come in contact with. But I would say a good once a week I stare aimlessly at my computer screen and wonder what the fuck I did in a previous life. Every now and then the phone's harsh ring invokes a momentary nightmare like a Pavlovian reaction. Occasionally I feel like having a drink in the late afternoon only to immediately feel guilt because the bar associations are on a rampage of sniffing out alcoholism like McGruff the Rabid Crime Dog.
I look at people my age who have less intellect and less impressive credentials who seem to be doing fine and more liberated from industry-wide black holes. Why did I take this road and not that one? It's difficult to know whether the grass is truly greener, but I know that grass is less burdened with debt and that I can work harder and smarter than average at tending it. Survey says I'm not the only one.
Simkovic & McIntyre's work systematically debunks any such thoughts. The great lie is that nuanced people and professions can be reduced to monotonous averages and that law school is either a wise investment or it is not in purely economic terms. As a lowly liberal artist, I was destined to make penurious wages. Law school was my salvation, other educational alternatives or putting sweat equity into a business be damned. My alternative realities are little more than economic and social pornography. In a way, it's a fatalistic view that removes the input of any decision save whether to go to law school or not. White, black, dumb, smart, ugly, beautiful. Just digits on an enrollment sheet.
Of course, Simkovic & McIntyre found a market already primed to believe in its hokey premium model. When looking at schools and law school as a whole, did we not compare our present salaries to the advertised ones and titter at the thought of an easy tripling? Law schools have been milking the premium angle, at least on an implied basis, to cash-starved 23-year-olds for decades. Simkovic & McIntyre merely found a way, post-recession, post-fixed-stats, to capitalize on that mode of thought with a regression analysis of gobbledygook horseshit.
In reality, very little of life's finer aspects come from a strictly economic evaluation of fungible people. A similar great lie, not explicitly discussed in Simkovic & McIntyre's work, is that many of us went into the law, at least in part, because we saw a rational, meritocratic profession that traded in truth and justice and rewarded intellect and hard work. The crushing blow of law, perhaps, comes from a mismatch between the perceived ex ante value of such traits and the actual (negative) value of such traits ex post.
Perhaps, then, Simkovic & McIntyre's work is a greater work of propaganda than even the best shills have given it credit. Not only do they play on the idea that legal careers can be evaluated (poorly) using (kindergarten-level) econometrics, they implicitly ignore all of the other items beyond money that make a career worthwhile and satisfying.
In tricking us all into arguing over financial prosperity or non-dischargable ruin based on a law degree premium, Simkovic & McIntyre have convinced us to ignore all of the other reasons being a lawyer creates unhappy careers at a rate faster than almost any other profession. Sure, Economic Value is unscientific hogwash on its face, but what isn't written at all is that law may be toxic for reasons other than cold hard cash (tempered with modest debt) and, for certain completely normal personality types, law school is unsafe at any price.
The law, in many ways, is about understanding nuance where others see similarity. In looking at multiple generations of diverse law school attendees with varying intellectual abilities and backgrounds, Simkovic & McIntyre's approach sees not nuance, but a big bucket of ageless interchangable lawyers unconcerned with justice or delaying starting a family. From a certain angle, not only is the work rubbish and hostile to the virtues of the scientific method, its assumptions are insulting to several core belief systems underpinning American law while still telling people using fancy numbers 'n' such to go to law school.
That anyone cited this ludicrous monstrosity of light teaching loads beyond patting Simkovic & McIntyre on the metaphorical forehead and telling them to try again is a sign of certain mastery. That they got the entire full-time staff of the Law School Truth Center to devote a nine-part series to their work?
That's genius.
Saturday, July 30, 2016
Wednesday, July 27, 2016
Restless Genius: Repeat Repeat Repeat (Part VIII)
We're almost done. Unlike Simkovic & McIntyre...
It's a cliche that academics, even brilliant ones, repeat themselves and merely do variations on a certain idea or theme. Simkovic & McIntyre are no different. Since publishing Economic Value, the dynamic duo has put out a joint review of Tamanaha's Failing Law Schools, an article called Timing Law School, a Simkovic-only screed on valuing higher education called A Value Added Perspective on Higher Education, and most recently (March 2016) a piece called Value of a Law Degree by College Major.
In their review of Tamanaha's work, they somehow make it to page six before discussing their own research, although the first few pages are not without merit (such as the brilliant straw-man that "Tamanaha hopes the government will prevent students from borrowing against their future earnings to pay for the cost of a legal education", the foot-note defense of regulatory capture, or that Tamahana's suggestion of legal residency programs ipso facto mean the taxpayer would fund them because medical school residences receive payments from Medicare).
The real brilliance is that discussion of their own work criticizes Tamahama for the exact same things Economic Value does. On page 11, for example, we have a criticism of mismatching terminal bachelor's holders with law degree holders followed by a criticism of "inconsistent definitions" from "different sources" with "different sampling and reporting biases." As you'll recall from our previous entries, Economic Value is unclear about what a "law degree" is, uses a bogus control group, and smashes together two unrelated governmental surveys to whoop ding ca-ching million dollar degree... for some group of people who entered practice a while ago.
Timing Law School is the follow-up album to Economic Value where Simkovic & McIntyre regurgitate their data, run some simulations, and conclude that the timing of when one goes to law school is unlikely to make law school a poor investment, only determining, maybe, how good of an investment law school is. Of course, once one concedes that there is variance in whatever law school earnings premium exists on a year-over-year basis and that large law firm employment is dynamic, one would also have to concede that the cumulative lifetime variance could be either negative or so extremely disproportional that it makes sense for some students (particularly at the high end) to hold on law school (or go immediately before a recession hits). A lawyer who graduated into the early 1970s recession has about as much relevance to present law graduates as '73 Chevelle sales have to next year's Chevy models.
But again, Simkovic & McIntyre stick to their guns like tried and true academic polemicists.
Simkovic's solo effort Value-Added is an essay that only drops Economic Value in a footnote but still manages to complement its legacy nicely by serving as an apologia for non-elite educational institutions. For example, Simkovic opens with a weak analogy contrasting schools to hospitals ("If hospitals were evaluated based purely on outcomes...") which, one supposes, compares the dying infirm to third-tier law applicants. The remainder of the essay proceeds by pretending to anticipate and knock down common sense arguments with brisk aplomb. Underemployment? Those folks just need degrees to correct for their disadvantages. Shit schools? Economic necessity. Market and regulatory failures? "Implausible."
Not implausible is that Simkovic & McIntyre will keep going, and indeed this past March put out Value of a Law Degree by College Major, a short article where they supplement the Economic Value data-set with information from the American Community Survey to gain some sort of post-2008 information, even though ACS does not identify who has a law degree (they even admit it's harder to tell who has a law degree than in SIPP, which we already noted was imperfect). It essentially repeats the college major conclusions from Economic Value, claiming that a law degree is a great option, even for STEM majors (in their terminal form, at least) and that a law degree "erases the disproportionately lower expected earnings from a liberal arts degree."
Again, Simkovic & McIntyre generally don't address alternative graduate degrees (that's for "future research") and don't seem to address the idea that law school (or other graduate options) may be a reason that many students who could excel in STEM choose liberal arts instead, i.e., that the college major decision is itself polluted by post-graduate options and not strictly a sorting of ability.
It's all bullshit with a glossy sheen of academese. If it looks like a duck, shoot it, baste it, and feast away.
At the end of the day, all good propagandists are bullshit artists who know that repeating something over and over again is the best way to get a message across, and with a big enough megaphone, truth or falsity become largely irrelevant. This law governs everything from toothpaste sales to world leadership. We have a major party presidential candidate who has bullshat his way to a Republican nomination.
The more Simkovic & McIntyre repeat their nonsense, the longer their track record becomes. Flaws in their original research are swept under the rug. Media outlets only read the conclusions, anyway. Enough bullshit conclusions stacked on top of each other and you've got a textbook of complete bullshit masquerading as the truth.
Genius is not a one-hit wonder. Genius never stops producing and never stops touring. For Simkovic & McIntyre, I would bet the ingenious output is by no means concluded. They won't stop until you're as convinced of their genius as I am.
It's a cliche that academics, even brilliant ones, repeat themselves and merely do variations on a certain idea or theme. Simkovic & McIntyre are no different. Since publishing Economic Value, the dynamic duo has put out a joint review of Tamanaha's Failing Law Schools, an article called Timing Law School, a Simkovic-only screed on valuing higher education called A Value Added Perspective on Higher Education, and most recently (March 2016) a piece called Value of a Law Degree by College Major.
In their review of Tamanaha's work, they somehow make it to page six before discussing their own research, although the first few pages are not without merit (such as the brilliant straw-man that "Tamanaha hopes the government will prevent students from borrowing against their future earnings to pay for the cost of a legal education", the foot-note defense of regulatory capture, or that Tamahana's suggestion of legal residency programs ipso facto mean the taxpayer would fund them because medical school residences receive payments from Medicare).
The real brilliance is that discussion of their own work criticizes Tamahama for the exact same things Economic Value does. On page 11, for example, we have a criticism of mismatching terminal bachelor's holders with law degree holders followed by a criticism of "inconsistent definitions" from "different sources" with "different sampling and reporting biases." As you'll recall from our previous entries, Economic Value is unclear about what a "law degree" is, uses a bogus control group, and smashes together two unrelated governmental surveys to whoop ding ca-ching million dollar degree... for some group of people who entered practice a while ago.
Timing Law School is the follow-up album to Economic Value where Simkovic & McIntyre regurgitate their data, run some simulations, and conclude that the timing of when one goes to law school is unlikely to make law school a poor investment, only determining, maybe, how good of an investment law school is. Of course, once one concedes that there is variance in whatever law school earnings premium exists on a year-over-year basis and that large law firm employment is dynamic, one would also have to concede that the cumulative lifetime variance could be either negative or so extremely disproportional that it makes sense for some students (particularly at the high end) to hold on law school (or go immediately before a recession hits). A lawyer who graduated into the early 1970s recession has about as much relevance to present law graduates as '73 Chevelle sales have to next year's Chevy models.
But again, Simkovic & McIntyre stick to their guns like tried and true academic polemicists.
Simkovic's solo effort Value-Added is an essay that only drops Economic Value in a footnote but still manages to complement its legacy nicely by serving as an apologia for non-elite educational institutions. For example, Simkovic opens with a weak analogy contrasting schools to hospitals ("If hospitals were evaluated based purely on outcomes...") which, one supposes, compares the dying infirm to third-tier law applicants. The remainder of the essay proceeds by pretending to anticipate and knock down common sense arguments with brisk aplomb. Underemployment? Those folks just need degrees to correct for their disadvantages. Shit schools? Economic necessity. Market and regulatory failures? "Implausible."
Not implausible is that Simkovic & McIntyre will keep going, and indeed this past March put out Value of a Law Degree by College Major, a short article where they supplement the Economic Value data-set with information from the American Community Survey to gain some sort of post-2008 information, even though ACS does not identify who has a law degree (they even admit it's harder to tell who has a law degree than in SIPP, which we already noted was imperfect). It essentially repeats the college major conclusions from Economic Value, claiming that a law degree is a great option, even for STEM majors (in their terminal form, at least) and that a law degree "erases the disproportionately lower expected earnings from a liberal arts degree."
Again, Simkovic & McIntyre generally don't address alternative graduate degrees (that's for "future research") and don't seem to address the idea that law school (or other graduate options) may be a reason that many students who could excel in STEM choose liberal arts instead, i.e., that the college major decision is itself polluted by post-graduate options and not strictly a sorting of ability.
It's all bullshit with a glossy sheen of academese. If it looks like a duck, shoot it, baste it, and feast away.
At the end of the day, all good propagandists are bullshit artists who know that repeating something over and over again is the best way to get a message across, and with a big enough megaphone, truth or falsity become largely irrelevant. This law governs everything from toothpaste sales to world leadership. We have a major party presidential candidate who has bullshat his way to a Republican nomination.
The more Simkovic & McIntyre repeat their nonsense, the longer their track record becomes. Flaws in their original research are swept under the rug. Media outlets only read the conclusions, anyway. Enough bullshit conclusions stacked on top of each other and you've got a textbook of complete bullshit masquerading as the truth.
Genius is not a one-hit wonder. Genius never stops producing and never stops touring. For Simkovic & McIntyre, I would bet the ingenious output is by no means concluded. They won't stop until you're as convinced of their genius as I am.
Sunday, July 24, 2016
Genius Because Paper met Wall: Sheepskin Effects and Economic Value (Part VII)
We continue our travel down the open road of economic happiness by evaluating Simkovic & McIntyre's Economic Value research in multi-part depth. Like making love to a woman who's clinically insane and on crack, we keep going back. Not sure why, but we do. And it keeps getting better and better and better...
In Part VI, we took a deeper look at the overall mathematical model of Economic Value, discussed some of the basic concepts, and praised Simkovic & McIntyre's genius is presenting a human capital theory that departed from the Mincer formula and its adaptations for reasons that remain brilliantly opaque.
In this part, we're going to ask why they ignored an alternative explanatory theory from labor economics.
Economic Value is premised on a human capital theory, i.e., that investments in human capital (through education) increase earnings and that the primary variable is the time spent on investment. Simkovic & McIntyre propose, essentially that the investment in human capital of going to law school benefits the student whether they work as a lawyer or not.
An alternative, however, is a specific signalling theory, referred to in the literature as a sheepkin effect, whereby it's the degree that causes a rise in the market value of the labor rather than the actual time investment in education. Matt Leichter basically covers this argument as well as it can be, so I'll let him take over:
It's not like Card is the only person who points the "sheepskin effect" out, either. It has been widely studied. But like true geniuses, Simkovic & McIntyre totally side-step its application.
At this point, we have to ask ourselves exactly what Simkovic & McIntyre are arguing. Are they even arguing a traditional human capital theory, or are they arguing the JD is a signal to employers that this person is worth more? As Leichter more or less points out, they've effectively taken both sides and provide no clarity about their precise thesis. He circles back to the control group problem discussed in Part III:
But fuck it, it's genius. Simkovic & McIntyre don't need a clear population definition. They don't need an adequate sample. They don't need an adequate control group. They don't need a recent sample - it's 2008 somewhere, right? They don't need to situate their literature in the existing theoretical landscape or address alternative theories to explain the observed phenomena. They don't need to address any of Leichter's eleven points.
Fuck science. Go to law school.
In Part VI, we took a deeper look at the overall mathematical model of Economic Value, discussed some of the basic concepts, and praised Simkovic & McIntyre's genius is presenting a human capital theory that departed from the Mincer formula and its adaptations for reasons that remain brilliantly opaque.
In this part, we're going to ask why they ignored an alternative explanatory theory from labor economics.
Economic Value is premised on a human capital theory, i.e., that investments in human capital (through education) increase earnings and that the primary variable is the time spent on investment. Simkovic & McIntyre propose, essentially that the investment in human capital of going to law school benefits the student whether they work as a lawyer or not.
An alternative, however, is a specific signalling theory, referred to in the literature as a sheepkin effect, whereby it's the degree that causes a rise in the market value of the labor rather than the actual time investment in education. Matt Leichter basically covers this argument as well as it can be, so I'll let him take over:
Without explanation, the authors reject signaling theory, an alternative that appears in an article by David Card cited in footnote 4 (PDF) of “Economic Value.” Card’s article displays a chart, though outdated, showing that professional school graduates make significantly more money than the trend predicts. Card identifies this phenomenon as the “sheepskin effect,” which means the additional earnings degree-holders obtain over what they would have earned with just an additional year of schooling.[i] Given that most states require a law degree to practice law, the sheepskin effect, which doubles here as a “licensed professional effect,” is probably very pronounced. The utility of the 58,000th minute of law school that the ABA mandates is very high indeed.Because I think its incredibly important and is an another insurmountable nail in the non-genius evaluation of Economic Value coffin, I'll lazily copy the chart with Leichter's notation here:
It's not like Card is the only person who points the "sheepskin effect" out, either. It has been widely studied. But like true geniuses, Simkovic & McIntyre totally side-step its application.
At this point, we have to ask ourselves exactly what Simkovic & McIntyre are arguing. Are they even arguing a traditional human capital theory, or are they arguing the JD is a signal to employers that this person is worth more? As Leichter more or less points out, they've effectively taken both sides and provide no clarity about their precise thesis. He circles back to the control group problem discussed in Part III:
At this point, we should declare “Economic Value” a mistrial. The authors overeagerly tested the data without thinking through the theory discussed in their own citations. A “premium” cannot be measured until someone compares the earnings of law school graduates to the earnings of those who’ve completed all or nearly all the required law school coursework but didn’t graduate. If the earnings of both groups are the same, then the authors have a basis for claiming that law school increases people’s earnings based on the human capital theory.In any event, Simkovic & McIntyre know the research is out there suggesting that credentials - and not years of education - may be causing any observable shift in earnings. They don't even make an attempt to address this theory, which would mean that we might be able to essentially give out law degrees to gifted BA holders with the same effect.
But fuck it, it's genius. Simkovic & McIntyre don't need a clear population definition. They don't need an adequate sample. They don't need an adequate control group. They don't need a recent sample - it's 2008 somewhere, right? They don't need to situate their literature in the existing theoretical landscape or address alternative theories to explain the observed phenomena. They don't need to address any of Leichter's eleven points.
Fuck science. Go to law school.
Wednesday, July 20, 2016
Genius Off the Deep End: Math and Formulas and Such (Part VI)
We continue our exploration of Economic Value. This one dives more deeply into shallow mathematics - or is it takes a shallow look at deep mathematics? Who the hell knows? This week I'm savoring the million dollar premium in a low-cost rental in hick country drinking glorified furniture polish, so I'm not inclined to care.
Most academic writers try to work from existing streams of thought in a particular area. Such a task - particularly in an area like economics - often takes the work outside the realm of comprehensibility for the general reading public. Simkovic & McIntyre steer clear of firmly placing their work within existing literature on human capital theory, but at the same time keep the work appropriately out of reach of the most unsophisticated readers, walking a tightrope that is the hallmark of geniuses like Beethoven or Stephen King.
For example, despite being written for what is mostly a non-scientific audience (let's not kid ourselves, right?), Economic Value gives rather short shrift to explaining its methods in approachable terms. So let's start by looking at the basic statistical approach on a more introductory level. One example of a term that goes under-explained is "Ordinary Least Squares" ("OLS") The wikipedia entry gives a fair, if dense, explanation. It's a standard method of regression analysis in research.
In layman's terms, the goal of regression analysis generally is that where you have what's called a dependent variable (in this particular case earnings), you take all of the independent variable observations you have (in our case, whether there's a "law degree" and a bunch of other factors that apply to a person), and you try to find a line, represented by a mathematical formula, that best "fits" the data provided to explain the relationship between all those other variables (education, experience, skills, race, gender, etc.) and earnings. The ultimate design is that you find some formula where you can plug in the numbers for independent variables, even if not seen in data, and generate a predicted output.
In their data modelling, Simkovic & McIntyre use what's called an independent "dummy variable" in the formula for whether the person has a law degree or not. What this means is that when the person has only a bachelor's degree, the dummy variable is set to 0 and the dependent variable (earnings) can be derived from all the other explanatory variables (like years of experience or some variable to represent raw ability). When the person has a "law degree," their dummy variable is set to 1.
The coefficient that goes with the dummy variable is thus what they ultimately use to determine their earnings premium, although the precise translation from this coefficient to actual numbers isn't crystal clear to the lay reader.
Another phrase often used, and not thoroughly explained, is "log earnings." "Log" stands for "logarithm," which can be explained here for those of you who quit before that particular math class. Basically hit the log button on your calculator and then type an amount of annual earnings. Simkovic & McIntyre use log earnings instead of simple dollar-value earnings because there is a tradition in quantitative labor economics to use log earnings as a dependent variable (one can read the Card article cited in Economic Value for an explanation).
However, one brilliant thing that using "log earnings" instead of raw earnings is that it allows for large spans to be concealed in what otherwise appear to be short distances. For example, Figure 5 in Economic Value is ostensibly used to show that "[r]ecent premiums for young law graduates are within historical norms."
Yet, note that as we pointed out in Part 3, the 2008-2013 grouping has an incredibly small number of observations. Look at the differences in log earnings represented in Figure 5, and pay particular attention to the confidence intervals. Simkovic & McIntyre try to demonstrate that there's not much variance with an "average premium" of 0.56 across the observations. But the confidence intervals range from <0.2 in 2004-2007 and ~1.0 for 2000-2003. Moreover, because logarithms are inversely exponential, very small changes in log earnings actually mean large shifts when it comes to real numbers that people actually have to buy things with in real life. For example, the difference represented in Figure 5 between the 2000-2003 premium and the 2004-2007 premium is 0.29. Well, log 100000 - log 50000 = 0.3 or so. That's a lot of variance to claim anything is "within historical norms" or steady across a small number of observations.
But still, using log earnings and a linear regression is fairly well accepted in the economic literature. There isn't too much controversial (or ingenious) in the basic idea.
A better question, though, is what genius path led Simkovic & McIntyre to reject what appears to be an existing common approach in the literature to human capital theories? I ask this question not to criticize, but to comprehend.
For several decades, labor economists have used Mincer's Human Capital Earnings Function to evaluate the relationship between education, earnings, and experience. Card - again, cited by Simkovic & McIntyre - discusses this literature at length.
The basic Mincer Formula (which has been expanded in other literature) is this:
Log(earnings) = log(baseline earnings) + a(years of schooling) + b(work experience) + c(work experience)^2
As fatally flawed as their SIPP data is, Simkovic & McIntyre had everything they needed to calculate the added earnings of three JD years using this formula or some custom derivation from it appropriately modified to the "law degree" market.
For example, one of the most common problems identified with the Mincer equation is that log earnings may not have a linear relationship with years of education (which is what results in the Mincer formula for people who have no experience). In other words, certain years of training may "pay off" more than others in terms of developing human capital. This possibility will be discussed more in Part 7.
Several scholars (including at least one, Card, cited by Simkovic & McIntyre) have found a convex relationship between earnings and education. As Lemieux (2003) suggests upon reviewing Mincer (1997), log earnings may only be linear in a stable environment where labor demand meets the incoming labor supply, pointing out that log earnings was convex where the demand for skilled labor outpaced the existing supply. Lemieux suggests adding an additional polynomial to the base model, and other researchers have played around with numerous formulations. There are people doing integral calculus with this shit in the 2000s.
So what formula do Simkovic and McIntyre use?
Log(earnings) = a(dummy for law degree) + monomial control variables and constants
What's missing? Well, in short, they've seemingly done away with experience being a variable in their basic model despite it being a crucial variable in decades of labor economics' studying of the relationship between education and earnings. Whereas the Mincer formula would not only factor experience as a key variable in the evaluation of log earnings but factor it exponentially, Simkovic & McIntyre seem to take a more circuitous route to addressing the role of experience in lawyer earnings. They've also (by the lack of any polynomials) assumed log earnings and education have a fundamentally linear relationship when such fact remains an open question.
On one hand, the scant mention and departure from Mincer and his progeny is odd - almost like ignoring strongly persuasive authority from multiple federal appeals courts, but again we must understand what Simkovic & McIntyre are doing: simplifying the mystical and clarifying the sloppily obfuscated. So what if they started from scratch with a lousy dataset and ignored what might be the most important variable in explaining earnings in the labor economics literature?
That they took a deeper mathematical approach than prior studies alone should win them praise; that they managed to sidestep certain issues with a deft use of numbers should win them super-tenure.
Again, corrections are welcome. As a lawyer, and not an economist, statistician, or professor of law, my station is inferior and I would welcome a greater understanding of Simkovic & McIntyre's keen analysis.
Most academic writers try to work from existing streams of thought in a particular area. Such a task - particularly in an area like economics - often takes the work outside the realm of comprehensibility for the general reading public. Simkovic & McIntyre steer clear of firmly placing their work within existing literature on human capital theory, but at the same time keep the work appropriately out of reach of the most unsophisticated readers, walking a tightrope that is the hallmark of geniuses like Beethoven or Stephen King.
For example, despite being written for what is mostly a non-scientific audience (let's not kid ourselves, right?), Economic Value gives rather short shrift to explaining its methods in approachable terms. So let's start by looking at the basic statistical approach on a more introductory level. One example of a term that goes under-explained is "Ordinary Least Squares" ("OLS") The wikipedia entry gives a fair, if dense, explanation. It's a standard method of regression analysis in research.
In layman's terms, the goal of regression analysis generally is that where you have what's called a dependent variable (in this particular case earnings), you take all of the independent variable observations you have (in our case, whether there's a "law degree" and a bunch of other factors that apply to a person), and you try to find a line, represented by a mathematical formula, that best "fits" the data provided to explain the relationship between all those other variables (education, experience, skills, race, gender, etc.) and earnings. The ultimate design is that you find some formula where you can plug in the numbers for independent variables, even if not seen in data, and generate a predicted output.
In their data modelling, Simkovic & McIntyre use what's called an independent "dummy variable" in the formula for whether the person has a law degree or not. What this means is that when the person has only a bachelor's degree, the dummy variable is set to 0 and the dependent variable (earnings) can be derived from all the other explanatory variables (like years of experience or some variable to represent raw ability). When the person has a "law degree," their dummy variable is set to 1.
The coefficient that goes with the dummy variable is thus what they ultimately use to determine their earnings premium, although the precise translation from this coefficient to actual numbers isn't crystal clear to the lay reader.
Another phrase often used, and not thoroughly explained, is "log earnings." "Log" stands for "logarithm," which can be explained here for those of you who quit before that particular math class. Basically hit the log button on your calculator and then type an amount of annual earnings. Simkovic & McIntyre use log earnings instead of simple dollar-value earnings because there is a tradition in quantitative labor economics to use log earnings as a dependent variable (one can read the Card article cited in Economic Value for an explanation).
However, one brilliant thing that using "log earnings" instead of raw earnings is that it allows for large spans to be concealed in what otherwise appear to be short distances. For example, Figure 5 in Economic Value is ostensibly used to show that "[r]ecent premiums for young law graduates are within historical norms."
Yet, note that as we pointed out in Part 3, the 2008-2013 grouping has an incredibly small number of observations. Look at the differences in log earnings represented in Figure 5, and pay particular attention to the confidence intervals. Simkovic & McIntyre try to demonstrate that there's not much variance with an "average premium" of 0.56 across the observations. But the confidence intervals range from <0.2 in 2004-2007 and ~1.0 for 2000-2003. Moreover, because logarithms are inversely exponential, very small changes in log earnings actually mean large shifts when it comes to real numbers that people actually have to buy things with in real life. For example, the difference represented in Figure 5 between the 2000-2003 premium and the 2004-2007 premium is 0.29. Well, log 100000 - log 50000 = 0.3 or so. That's a lot of variance to claim anything is "within historical norms" or steady across a small number of observations.
But still, using log earnings and a linear regression is fairly well accepted in the economic literature. There isn't too much controversial (or ingenious) in the basic idea.
A better question, though, is what genius path led Simkovic & McIntyre to reject what appears to be an existing common approach in the literature to human capital theories? I ask this question not to criticize, but to comprehend.
For several decades, labor economists have used Mincer's Human Capital Earnings Function to evaluate the relationship between education, earnings, and experience. Card - again, cited by Simkovic & McIntyre - discusses this literature at length.
The basic Mincer Formula (which has been expanded in other literature) is this:
Log(earnings) = log(baseline earnings) + a(years of schooling) + b(work experience) + c(work experience)^2
As fatally flawed as their SIPP data is, Simkovic & McIntyre had everything they needed to calculate the added earnings of three JD years using this formula or some custom derivation from it appropriately modified to the "law degree" market.
For example, one of the most common problems identified with the Mincer equation is that log earnings may not have a linear relationship with years of education (which is what results in the Mincer formula for people who have no experience). In other words, certain years of training may "pay off" more than others in terms of developing human capital. This possibility will be discussed more in Part 7.
Several scholars (including at least one, Card, cited by Simkovic & McIntyre) have found a convex relationship between earnings and education. As Lemieux (2003) suggests upon reviewing Mincer (1997), log earnings may only be linear in a stable environment where labor demand meets the incoming labor supply, pointing out that log earnings was convex where the demand for skilled labor outpaced the existing supply. Lemieux suggests adding an additional polynomial to the base model, and other researchers have played around with numerous formulations. There are people doing integral calculus with this shit in the 2000s.
So what formula do Simkovic and McIntyre use?
Log(earnings) = a(dummy for law degree) + monomial control variables and constants
What's missing? Well, in short, they've seemingly done away with experience being a variable in their basic model despite it being a crucial variable in decades of labor economics' studying of the relationship between education and earnings. Whereas the Mincer formula would not only factor experience as a key variable in the evaluation of log earnings but factor it exponentially, Simkovic & McIntyre seem to take a more circuitous route to addressing the role of experience in lawyer earnings. They've also (by the lack of any polynomials) assumed log earnings and education have a fundamentally linear relationship when such fact remains an open question.
On one hand, the scant mention and departure from Mincer and his progeny is odd - almost like ignoring strongly persuasive authority from multiple federal appeals courts, but again we must understand what Simkovic & McIntyre are doing: simplifying the mystical and clarifying the sloppily obfuscated. So what if they started from scratch with a lousy dataset and ignored what might be the most important variable in explaining earnings in the labor economics literature?
That they took a deeper mathematical approach than prior studies alone should win them praise; that they managed to sidestep certain issues with a deft use of numbers should win them super-tenure.
Again, corrections are welcome. As a lawyer, and not an economist, statistician, or professor of law, my station is inferior and I would welcome a greater understanding of Simkovic & McIntyre's keen analysis.
Sunday, July 17, 2016
Genius Interjected: What the Hell is this NELS Stuff? (Part V)
We continue our exploration of Simkovic & McIntyre's Economic Value. You could call this a special summer serial, but it's actually just an LSTC that's too lazy to come up with stories in the deadest month of the year.
By this point, we have established that SIPP is a poor method through which to obtain samples of the JD-holding population in the present for useful evaluation. We are not exactly novel in pointing some of these things out.
But genius is unflappable. When it has one clunky government survey that's designed for an entirely different purpose and insufficient for the conclusions at hand, genius doubles down and finds a totally unrelated clunky second one from which to draw different, related conclusions.
In Economic Value Simkovic & McIntyre suddenly invoke the National Educational and Longitudinal Study ("NELS") of '88. What the Dept. of Ed did with NELS is start with a survey of ~25k 8th graders (or thereabouts) in 1987-88 and then it used a variety of methods to follow them up to the year 2000.
Note the obvious limitations here:
To some extent, then, Economic Value - whose authors wanted to look at the lifetime JD premium and who stress that first-year salaries are not reflective - seem to suddenly use "terminal" earnings reports from 27-year olds (the last panel of NELS was done in 2000) to calculate differences in future earnings among various factors. So there's no accounting for the teachers with 30 service years and a fat pension or the restauranteur who toils for a decade and then opens their own successful bar, no consideration that these folks follow that mythical, exponential career arc that eventually shoots $35k new lawyers to the economic heavens.
They then used those numbers - from a not-necessarily-representative sample derived from one segment of their target population - to compare it to the regression results from their SIPP data to conclude that the "ability sorting" variables they could identify in the NELS were not significant in changing their SIPP-based conclusions.
In other words, they used NELS (again, an unrelated survey from one cohort) to rule out a variety of alternative theories about the "premium" identified in the SIPP pool coming from sources or characteristics other than a JD - a time span that covers like 50 years of JDs and evaluates nothing at all after 2009.
Given that NELS is limited to one age group, its use to make any relevant conclusions at all about their other non-representative sample pooled from SIPP is stacking bullshit on top of bullshit and coating it with a thick layer of bullshit.
In other words, it's art. When a mediocre artist or scholar has something discardable and unusable, they throw it away. But Simkovic & McIntyre refused to sweep their deformed scholarly child into the trash. Instead, like many of postmodernism's fine artists faced with trashy art, they made trash itself art. Likewise, piling junk statistics onto junk statistics creates a masterpiece of legal scholarship.
By this point, we have established that SIPP is a poor method through which to obtain samples of the JD-holding population in the present for useful evaluation. We are not exactly novel in pointing some of these things out.
But genius is unflappable. When it has one clunky government survey that's designed for an entirely different purpose and insufficient for the conclusions at hand, genius doubles down and finds a totally unrelated clunky second one from which to draw different, related conclusions.
In Economic Value Simkovic & McIntyre suddenly invoke the National Educational and Longitudinal Study ("NELS") of '88. What the Dept. of Ed did with NELS is start with a survey of ~25k 8th graders (or thereabouts) in 1987-88 and then it used a variety of methods to follow them up to the year 2000.
Note the obvious limitations here:
- It appears to be one age cohort who would have started graduating law school circa 1999, and therefore entirely non-representative of the population in SIPP, or in reality absent some assumptions that don't work.
- When one reads the associated guides and materials, it becomes apparent it's not even a truly representative sample of the population of students for that age group.
- The sources of information in the survey are not even consistent.
- Using this argument/analysis is basically an admission that inputs matter significantly in the process and that the idea of a unified "premium" for law school is inane.
- It seems to ignore that the designation of majors/fields of study is inherently corrupted by the presence of post-graduate options. A student who might otherwise study engineering and be fully capable to completing the engineering curriculum (leading to engineer money) may opt to major in liberal arts and study business or law; grouping such a student with the terminal bachelor's liberal arts students is disingenuous. In fact, the primary study that almost certainly inspired using longitudinal studies and evaluating "ability sorting" (http://public.econ.duke.edu/~psarcidi/arcidimetrics.pdf, cited in Economic Value) focused on people who graduated in the 1970s and last reported earnings in 1986 and ignores the role of graduate school, which is likely more forgivable for students from the early 70s than students who went to school in the late 90s.
To some extent, then, Economic Value - whose authors wanted to look at the lifetime JD premium and who stress that first-year salaries are not reflective - seem to suddenly use "terminal" earnings reports from 27-year olds (the last panel of NELS was done in 2000) to calculate differences in future earnings among various factors. So there's no accounting for the teachers with 30 service years and a fat pension or the restauranteur who toils for a decade and then opens their own successful bar, no consideration that these folks follow that mythical, exponential career arc that eventually shoots $35k new lawyers to the economic heavens.
They then used those numbers - from a not-necessarily-representative sample derived from one segment of their target population - to compare it to the regression results from their SIPP data to conclude that the "ability sorting" variables they could identify in the NELS were not significant in changing their SIPP-based conclusions.
In other words, they used NELS (again, an unrelated survey from one cohort) to rule out a variety of alternative theories about the "premium" identified in the SIPP pool coming from sources or characteristics other than a JD - a time span that covers like 50 years of JDs and evaluates nothing at all after 2009.
Given that NELS is limited to one age group, its use to make any relevant conclusions at all about their other non-representative sample pooled from SIPP is stacking bullshit on top of bullshit and coating it with a thick layer of bullshit.
In other words, it's art. When a mediocre artist or scholar has something discardable and unusable, they throw it away. But Simkovic & McIntyre refused to sweep their deformed scholarly child into the trash. Instead, like many of postmodernism's fine artists faced with trashy art, they made trash itself art. Likewise, piling junk statistics onto junk statistics creates a masterpiece of legal scholarship.
Thursday, July 14, 2016
Genius Uncontrolled, or, The Control Group Problem (Part IV)
This continues our series on Simkovic & McIntyre's still-cited masterpiece Economic Value as part of a pre-set July publication schedule. For the teeming tens of thousands of regular readers, a reminder that new content (or should I say "content") will return in August.
In Economic Value, Simkovic & McIntyre stress that they are looking for a premium provided by the "law degree" rather than an absolute return on investment or absolute earnings. That's a fine theoretical approach ("much like law school, it works on paper!"), except there are a number of practical problems with their chosen methods.
First, a "law degree" costs time and money. Lots of it. Simkovic & McIntyre address the return on investment part in Economic Value, but they seem to understate the costs associated with acquiring the premium. For the "best and brightest," three years of a working life at the start of a career will easily eat $120,000 or more of nominal earnings that can be invested for longer terms than money made later in life.
Second, they chose a terrible baseline. Simkovic & McIntyre gauge the premium of the "law degree" by comparisons between the "law degree" and what they call the "terminal bachelor's holder."
What they are trying to do is use a "control group" to evaluate the effect of a certain variable (in this case "law degree") within a broader population. For the lawyer who hates mathematics and science like an uninsured tortfeaser, a control group is basically an attempt to find something that is comparable and otherwise isolates the variable seeking to be studied to rule out any other explanations in advance. Assume you want to study the effectiveness of a particular brand of pesticide on apple trees. To do that, you would need a group of apple trees that do not have the pesticide, but otherwise match up well in other variables (like age of tree, climate, rainfall, soil chemistry, insects and other predators present, etc.). The more all other possible variables match up to a group that does have the pesticide, the more one can conclude that any differences between the two studied groups (the control group and the experimental group) are due to the variable.
By analogy, if we're studying the positive effect of a "law degree," we need to find a group of students who are otherwise identical to the group of people who landed "law degrees," but chose other paths instead.
The comparison between terminal bachelor's holders as a control group and "law degree" holders as the experimental group is therefore bogus (obviously, and, frankly, making such a comparison appears to lack complete intellectual honesty) for several reasons that immediately come to mind.
At this point, we're only on Part IV of our series and we've identified a definitional problem with the population/sub-population being studied, a sample size problem for the sub-populations divided by yearly cohorts, and a problem with a control group that makes no intuitive sense and leaves open a variety of variables that may be affecting the study results (for example, that the smartest terminal BA holders do just as well as those with law degrees?).
I think that what Simkovic & McIntyre did in this case is remarkably courageous and stupendously brilliant. Knowing that most lawyers are stupid when it comes to math and science, and knowing also that a law degree makes quite a bit of money over the long haul, Simkovic & McIntyre found an insightful and ingenious way to shit on paper and prove statistically that a law degree was a move of champions. The ends justify the means, after all. Cavalier statistical mis-steps be damned - this paper does a public service.
Again, that is the hallmark of true genius. Someone should cut these gentlemen a MacArthur grant yesterday.
In Economic Value, Simkovic & McIntyre stress that they are looking for a premium provided by the "law degree" rather than an absolute return on investment or absolute earnings. That's a fine theoretical approach ("much like law school, it works on paper!"), except there are a number of practical problems with their chosen methods.
First, a "law degree" costs time and money. Lots of it. Simkovic & McIntyre address the return on investment part in Economic Value, but they seem to understate the costs associated with acquiring the premium. For the "best and brightest," three years of a working life at the start of a career will easily eat $120,000 or more of nominal earnings that can be invested for longer terms than money made later in life.
Second, they chose a terrible baseline. Simkovic & McIntyre gauge the premium of the "law degree" by comparisons between the "law degree" and what they call the "terminal bachelor's holder."
What they are trying to do is use a "control group" to evaluate the effect of a certain variable (in this case "law degree") within a broader population. For the lawyer who hates mathematics and science like an uninsured tortfeaser, a control group is basically an attempt to find something that is comparable and otherwise isolates the variable seeking to be studied to rule out any other explanations in advance. Assume you want to study the effectiveness of a particular brand of pesticide on apple trees. To do that, you would need a group of apple trees that do not have the pesticide, but otherwise match up well in other variables (like age of tree, climate, rainfall, soil chemistry, insects and other predators present, etc.). The more all other possible variables match up to a group that does have the pesticide, the more one can conclude that any differences between the two studied groups (the control group and the experimental group) are due to the variable.
By analogy, if we're studying the positive effect of a "law degree," we need to find a group of students who are otherwise identical to the group of people who landed "law degrees," but chose other paths instead.
The comparison between terminal bachelor's holders as a control group and "law degree" holders as the experimental group is therefore bogus (obviously, and, frankly, making such a comparison appears to lack complete intellectual honesty) for several reasons that immediately come to mind.
- A "law degree" isn't just a extra sheet of paper. It costs three years to acquire with focused study in a particular field. Thus, the proper control group would be bachelor's degree plus three years. It isn't like "law degree" holders can be compared with 22-year olds. The oranges-to-oranges comparison would be bachelor's degree holders who are twenty-five or older - people who could actually have a "law degree" from a raw time-elapsed perspective. Indeed, in Economic Value, it appears Simkovic & McIntyre included all bachelor's degree holders as-of the start date of the survey, but excluded those on their way to getting a law degree (as naturally someone in law school would not report having completed a professional degree in law). That means people who are 23 and for whom it would be mathematically impossible to have a law degree are in the control group. That's bad.
- People who are eligible for a "law degree" have to be admitted to law school first. The requirements for law school (much less earning a JD) are more stringent than having a terminal bachelor's degree, and they likely always will be even as applicant standards plummet. So why are the two being compared? Simkovic & McIntyre hint at this with their discussion on ability sorting, but ultimately fail to use any reasonable method of using admissions screening to build a control group. Traditionally, law schools have only picked from the top third or top half of bachelor's degree holders. The proper control group here would be bachelor's degree holders who could have reasonably been admitted to law school. A "law degree" holder who graduated in 2006 has no business being compared to a 2003 college graduate who had a 2.3 GPA from Backwater State and a 134 on the LSAT. Yet, Simkovic & McIntyre apparently have no compunction making the comparison.
- People who don't go to law school can often get other forms of advanced study. A JD takes longer than an MBA or numerous other masters degrees that could boost earnings just as much, if not more. Limiting the "control group" to folks who terminated with a bachelor's degree is arguably methodologically dishonest, as it excludes arguably the most comparable educational options - and one that a lot of JDs would actually have done had they not gone to law school. I understand excluding medical and dental students, but a proper control group shouldn't be limited to just terminal degree holders. SIPP identifies master's holders in non-law areas. Why on Earth are these excluded from the control group? If anything, MBA students would provide a significantly better baseline than terminal bachelor's holders.
At this point, we're only on Part IV of our series and we've identified a definitional problem with the population/sub-population being studied, a sample size problem for the sub-populations divided by yearly cohorts, and a problem with a control group that makes no intuitive sense and leaves open a variety of variables that may be affecting the study results (for example, that the smartest terminal BA holders do just as well as those with law degrees?).
I think that what Simkovic & McIntyre did in this case is remarkably courageous and stupendously brilliant. Knowing that most lawyers are stupid when it comes to math and science, and knowing also that a law degree makes quite a bit of money over the long haul, Simkovic & McIntyre found an insightful and ingenious way to shit on paper and prove statistically that a law degree was a move of champions. The ends justify the means, after all. Cavalier statistical mis-steps be damned - this paper does a public service.
Again, that is the hallmark of true genius. Someone should cut these gentlemen a MacArthur grant yesterday.
Sunday, July 10, 2016
Genius is a Minority, or The Too-Small, Non-Representative Sample (Part III)
This is part 3 of our examination of Simkovic & McIntyre's Economic Value, perhaps the greatest law review article ever saved to a hard drive, the legal scholarship equivalent of James Joyce's Ulysses, becoming more and more intellectually rewarding upon further examination.
Repeatedly, Simkovic & McIntyre have defended the use of SIPP data, standing behind their methodology as all steadfast geniuses do. To their credit, using SIPP is the sort of move that geniuses make when everyone else is still adding up ABA 509 reports like chumps.
However, using SIPP data in regards to building a sample of the "law degree" population has two major problems: first, the way Simkovic & McIntyre have used SIPP causes their sample to be a non-random, non-representative sample of the law graduate population as a whole; and second, SIPP itself is way, way too small to generate conclusions for sub-samples based on later graduation years (as would be required to put forth any viable conclusions about changes over time in the fortunes of law graduates, which Simkovic & McIntyre purport to do).
Let's start by discussing how SIPP works. SIPP is a (usually) four-year study designed to review income and government program participation. It surveys somewhere between 14 and 52 thousand households in a series of "waves", with a recall of four months to each household. There were panels done in 1996, 2001, 2004, and 2008.
The questions about "degree" and "law" are asked early (Wave 2), so Simkovic & McIntyre built their sample using people who had a "law degree" on or before roughly 1996, 2001, 2004, and 2008, respectively (although it does appear some individuals reported a law degree in 2009).
Keep in mind that each panel of SIPP draws its sample from a nationwide population, and each sample is considered individually representative and randomly-drawn for the population at the time. Immediately, one should note that they are not (nor could they be) representative in the aggregate. If you pool the independently drawn samples together when they were conducted at different times and the population state changed from the first time to the second, you no longer have a representative sample.
As a more practical example, consider a high school where, for four years, you randomly select students from the entire student body. Your final sample is not representative of the high school population at any given time because you're going to have oversampled from the freshmen class when the survey began, and relatively undersampled from the first senior class and the last freshman class.
Similarly, consider a person who graduated law school in 1994. They might have been chosen as one who had a "law degree" in any of the four SIPP panels used by Simkovic & McIntyre. Meanwhile, a 2007 graduate is only possible to be selected in one of the SIPP panels: 2008's. So, assuming a constant flow of graduates, without any sort of normalization or procedures to control for the year one received a law degree, Simkovic & McIntyre's sample of "law degree" holders is going to include approximately two times as many 1993 graduates as 2003 graduates.
If you don't believe me, Simkovic has a chart apparently used in PowerPoint presentations that shows a bell curve over time. In reality, however, the number of law degrees generally increased year over year during the entire duration of the sample time. That Simkovic & McIntyre's sample looks nothing like a randomized sample of lawyers as-of 2008 would look should be enough by itself to invalidate the survey entirely.
The low sample size for more recent classes causes serious problems. Consider the 2007 "law degree" graduates. How many are included in Simkovic's sample? About 10. How many existed in real life? About 43,920.
Would you ever draw a conclusion about the economic fortunes of 43,920 people based on 10?
Thankfully, statistics has an answer to this issue that can provide some guidance. With a sample size of only 10 people in a population of 43,920, the confidence interval is a mere 31 points (!) at the 95% confidence level. Without belaboring the meaning of confidence intervals, that's a massive amount of uncertainty that no researcher on the planet would present as viable proof of anything.
To draw any sort of reasonable conclusions about the 2007 graduating class as a separate sub-population (i.e., to compare it with other sub-populations), one would need to randomly sample around 380 graduates. Even at "low" research standards (90% confidence level and 10% acceptable error), you would have to survey between 60-70 graduates from that graduating year cohort.
In fact, for most of the years featured in Simkovic & McIntyre's sample, there are simply too few observations to normally draw any sort of conclusions about the sub-population of graduates for those years taken separately. While the sample size is large enough to generate usable conclusions for the entire population independent of year graduated (well, if it were randomly drawn), such conclusions would be inescapably meaningless in a practical sense.
This is a classic subsampling problem, made all the more worse because Simkovic & McIntyre didn't have any samples at all drawn from 2010 and beyond despite those being the most relevant years for people applying to law school now.
Once we've shown the sample is non-representative of the subject population, and, in any event, too small to draw conclusions about sub-populations when broken down over time, its utility as a viable tool of prediction for independently-defined subgroups is little more than academic bullshit.
At its best, Economic Value only shows that a non-representative group of law graduates did better than a non-representative group of terminal bachelor's holders in the late 20th century. Would such a conclusion mean anything? Even be worth publishing?
For most of us, the answer would clearly be "no," and a rather resounding answer at that. At this point, absent being able to make any bona fide conclusions about sub-populations, with only tepid conclusions even remotely possible, the survey would become an aborted idea of the type that good researchers not gung-ho on publishing for publishing's sake have all the time.
But not Simkovic & McIntyre. Like the best of our pro se litigants, believing in their convictions without reservation or refined concession of error, they pressed forward. They doubled down, and published their research anyway, small, non-representative sample size be damned. They had biases to confirm, after all, and confirm biases they would. That type of atypical thinking is a hallmark characteristic of both lunacy and genius, and I think we all know that Simkovic & McIntyre clearly fall into the latter camp.
As stated previously, the author completely disclaims any alleged facts in this article, advises the audience that there is a high chance of error due to the author's third-tier education, and encourages the wise reader to forward a link or article that satisfactorily addresses or corrects the above.
Repeatedly, Simkovic & McIntyre have defended the use of SIPP data, standing behind their methodology as all steadfast geniuses do. To their credit, using SIPP is the sort of move that geniuses make when everyone else is still adding up ABA 509 reports like chumps.
However, using SIPP data in regards to building a sample of the "law degree" population has two major problems: first, the way Simkovic & McIntyre have used SIPP causes their sample to be a non-random, non-representative sample of the law graduate population as a whole; and second, SIPP itself is way, way too small to generate conclusions for sub-samples based on later graduation years (as would be required to put forth any viable conclusions about changes over time in the fortunes of law graduates, which Simkovic & McIntyre purport to do).
Let's start by discussing how SIPP works. SIPP is a (usually) four-year study designed to review income and government program participation. It surveys somewhere between 14 and 52 thousand households in a series of "waves", with a recall of four months to each household. There were panels done in 1996, 2001, 2004, and 2008.
The questions about "degree" and "law" are asked early (Wave 2), so Simkovic & McIntyre built their sample using people who had a "law degree" on or before roughly 1996, 2001, 2004, and 2008, respectively (although it does appear some individuals reported a law degree in 2009).
Keep in mind that each panel of SIPP draws its sample from a nationwide population, and each sample is considered individually representative and randomly-drawn for the population at the time. Immediately, one should note that they are not (nor could they be) representative in the aggregate. If you pool the independently drawn samples together when they were conducted at different times and the population state changed from the first time to the second, you no longer have a representative sample.
As a more practical example, consider a high school where, for four years, you randomly select students from the entire student body. Your final sample is not representative of the high school population at any given time because you're going to have oversampled from the freshmen class when the survey began, and relatively undersampled from the first senior class and the last freshman class.
Similarly, consider a person who graduated law school in 1994. They might have been chosen as one who had a "law degree" in any of the four SIPP panels used by Simkovic & McIntyre. Meanwhile, a 2007 graduate is only possible to be selected in one of the SIPP panels: 2008's. So, assuming a constant flow of graduates, without any sort of normalization or procedures to control for the year one received a law degree, Simkovic & McIntyre's sample of "law degree" holders is going to include approximately two times as many 1993 graduates as 2003 graduates.
If you don't believe me, Simkovic has a chart apparently used in PowerPoint presentations that shows a bell curve over time. In reality, however, the number of law degrees generally increased year over year during the entire duration of the sample time. That Simkovic & McIntyre's sample looks nothing like a randomized sample of lawyers as-of 2008 would look should be enough by itself to invalidate the survey entirely.
The low sample size for more recent classes causes serious problems. Consider the 2007 "law degree" graduates. How many are included in Simkovic's sample? About 10. How many existed in real life? About 43,920.
Would you ever draw a conclusion about the economic fortunes of 43,920 people based on 10?
Thankfully, statistics has an answer to this issue that can provide some guidance. With a sample size of only 10 people in a population of 43,920, the confidence interval is a mere 31 points (!) at the 95% confidence level. Without belaboring the meaning of confidence intervals, that's a massive amount of uncertainty that no researcher on the planet would present as viable proof of anything.
To draw any sort of reasonable conclusions about the 2007 graduating class as a separate sub-population (i.e., to compare it with other sub-populations), one would need to randomly sample around 380 graduates. Even at "low" research standards (90% confidence level and 10% acceptable error), you would have to survey between 60-70 graduates from that graduating year cohort.
In fact, for most of the years featured in Simkovic & McIntyre's sample, there are simply too few observations to normally draw any sort of conclusions about the sub-population of graduates for those years taken separately. While the sample size is large enough to generate usable conclusions for the entire population independent of year graduated (well, if it were randomly drawn), such conclusions would be inescapably meaningless in a practical sense.
This is a classic subsampling problem, made all the more worse because Simkovic & McIntyre didn't have any samples at all drawn from 2010 and beyond despite those being the most relevant years for people applying to law school now.
Once we've shown the sample is non-representative of the subject population, and, in any event, too small to draw conclusions about sub-populations when broken down over time, its utility as a viable tool of prediction for independently-defined subgroups is little more than academic bullshit.
At its best, Economic Value only shows that a non-representative group of law graduates did better than a non-representative group of terminal bachelor's holders in the late 20th century. Would such a conclusion mean anything? Even be worth publishing?
For most of us, the answer would clearly be "no," and a rather resounding answer at that. At this point, absent being able to make any bona fide conclusions about sub-populations, with only tepid conclusions even remotely possible, the survey would become an aborted idea of the type that good researchers not gung-ho on publishing for publishing's sake have all the time.
But not Simkovic & McIntyre. Like the best of our pro se litigants, believing in their convictions without reservation or refined concession of error, they pressed forward. They doubled down, and published their research anyway, small, non-representative sample size be damned. They had biases to confirm, after all, and confirm biases they would. That type of atypical thinking is a hallmark characteristic of both lunacy and genius, and I think we all know that Simkovic & McIntyre clearly fall into the latter camp.
As stated previously, the author completely disclaims any alleged facts in this article, advises the audience that there is a high chance of error due to the author's third-tier education, and encourages the wise reader to forward a link or article that satisfactorily addresses or corrects the above.
Thursday, July 7, 2016
Genius, Defined; or, What Is a Law Degree? (Part II)
This continues our exploration of Simkovic & McIntyre's Economic Value, a masterpiece of monumental thought leadership from two of the American northeast's leading educational institutions. If only Richard Epstein and Laurence Tribe taught in New Jersey, we'd have so much social justice and constitutional understanding, the justice system would literally grind to a halt to marvel, like Narcissus, at its own beauty.
"Genius" is a subjective term, absent some quantitative measurement like an intelligence quotient test that would omit at least half of what we consider genius and include some on-paper-geniuses that, in the common sense, do not qualify. As a result, studying "genius" in a population is quite challenging for researchers.
"Law degree," however, would seem like an objective term: a juris doctor degree from an ABA-accredited school.
This definition is essential to a study like Simkovic & McIntyre's. Not only does it ensure that a properly-configured population is being studied, it also allows for subsequent researchers to repeat Simkovic & McIntyre's observations. If scientist A discovers something amazing, he needs to leave a blueprint so that scientist B can follow.
However, in attempting to replicate and confirm Simkovic & McIntyre's Earth-shattering conclusions, one (read: the LSTC) runs into an immediate problem: What the heck do they mean by "law degree?"
This isn't a trick question or quixotic philosophical detour. The term is in the title, the abstract uses the phrase eight (8) times, and it's the main variable being studied. In common reporting on the study, the term "law degree" appears to be assumed synonymous with the juris doctor. (See, e.g., the ABA Journal write-up).
But here's the rub: Simkovic & McIntyre do not appear to have measured juris doctor holders.
The problem here is that the Census Bureau's Survey of Income and Program Participation ("SIPP"), used by Simkovic & McIntyre to build their sample set, does not appear to ask respondents whether they have a juris doctor. Instead, one of the SIPP topical modules (Wave 2, as viewed with the Census Bureau's DataFerret tool) asks respondents whether they have a "professional degree."
"Professional degree" does include the J.D as one example of a "professional degree," but there are two other categories that law graduates may possibly select outside of normal response error: master's and doctorate. What Simkovic & McIntyre appear to have done is cross-reference that answer with a separate question on the topical module asking what "field" the individual received that degree in.
In a footnote (fn 16 in this version, which may be a draft?) (see below), Simkovic & McIntyre explain that they required respondents to identify as having a "professional degree" in the field of "law." They elaborated that they excluded some unexplained number of people identifying as having a master's or doctorate or "imputed to be lawyers."
This approach may be both over-inclusive and under-inclusive in context. It's over-inclusive because there are other "professional" degrees in the field of law besides the juris doctor. Foreign-educated attorneys, for example, may rightly claim to have such degrees. Additionally, it is impossible to know if the sample is polluted with advanced paralegal degrees or others who have any of the myriad degrees that have spawned in the last twenty years.
It's under-inclusive because it's incredibly likely that some juris doctor holders checked a different box. Don't take my word for it; look at the actual numbers.
Looking at the 2008 SIPP responses and comparing people who selected "law" as their field of study in the Wave 2/topical module among various types of degrees identified, one sees the following:
Master's: 54
Professional School: 371
Doctorate: 67
By their explicit words, Simkovic & McIntyre apparently only included the 371 "professional" degree holders.
But what about the 67 people who identified having a "doctorate" in "law"? Are we assuming that none of these people have a juris doctor? Because I would imagine almost all of them do - it isn't like traditional PhD's in "law" are common (particularly because "social sciences" only has 33 and "liberal arts" only has 14 for the 2008 sample).
From a statistical purity standpoint, we have a big problem: people who almost certainly should be included are purposefully excluded by the study's apparent design. Not just 1 or 2%, but it's quite possible that 15% of the 2008 SIPP juris doctor holders were excluded as not having "law degrees."
And that's why the definition matters. When we ask law schools to collect data on their own graduates, we know those individuals have juris doctor degrees. If we exclude someone, it is because (in theory) we lack the data.
When Simkovic & McIntyre infer that a person has a "law degree" based on having a "professional" degree in "law" in census records, they introduce two layers of response error and use imprecise terminology.
The result is that the Economic Value may include people who don't have a juris doctor, almost certainly excludes people who do have a juris doctor, and tries to implicitly slide by on the premise that what they infer to be "law degrees" based on an unrelated governmental survey actually are people with juris doctor degrees.
Simkovic & McIntyre have repeatedly emphasized that other studies are over- or underinclusive and theirs hits the sweet spot:
When you consider that the tracking of lawyers includes only people who have a JD or the equivalent, or that standardized law school tracking already includes the exact, dead-on population, such a claim appears to be totally unsupportable bullshit in light of SIPP's limitations of definition.
That Simkovic & McIntyre - and numerous others - have persisted in using this study to purportedly support conclusions about the population of ABA-accredited juris doctor holders - even though the study's explanations of how it defined such a population appear to be unacceptably imprecise on their face - is a testament to Simkovic & McIntyre's slick genius.
Most scientists would never be able to pull off using a bad, imprecise data set as a superior means of study than using records sampled from the exact population. The most renowned scholars on Earth - Hawking, Dawkins, Andrew Fucking Wiles - would never dare pass off an imprecise definition as the one true authentic representation of a million-person population.
But Simkovic & McIntyre did. It may significantly alter the results of their (still-otherwise flawed) study; it may not have any effect at all. The fact that it exists without any real explanation shows that Simkovic & McIntyre clearly belong in the "genius" category regardless of how we define the term.
Note: there appear to be multiple versions of Economic Value circulating on the internet. If anyone is aware of a "final" version that addresses any issue raised in this post or any subsequent post, please comment or forward a link to my attention. In any event, the author expressly disclaims any purported factual allegations or implications in this or any other part of the series and states affirmatively that he/it is a moron of improper breeding and education and that readers should defer to the educated professorate to guide our thoughts.
"Genius" is a subjective term, absent some quantitative measurement like an intelligence quotient test that would omit at least half of what we consider genius and include some on-paper-geniuses that, in the common sense, do not qualify. As a result, studying "genius" in a population is quite challenging for researchers.
"Law degree," however, would seem like an objective term: a juris doctor degree from an ABA-accredited school.
This definition is essential to a study like Simkovic & McIntyre's. Not only does it ensure that a properly-configured population is being studied, it also allows for subsequent researchers to repeat Simkovic & McIntyre's observations. If scientist A discovers something amazing, he needs to leave a blueprint so that scientist B can follow.
However, in attempting to replicate and confirm Simkovic & McIntyre's Earth-shattering conclusions, one (read: the LSTC) runs into an immediate problem: What the heck do they mean by "law degree?"
This isn't a trick question or quixotic philosophical detour. The term is in the title, the abstract uses the phrase eight (8) times, and it's the main variable being studied. In common reporting on the study, the term "law degree" appears to be assumed synonymous with the juris doctor. (See, e.g., the ABA Journal write-up).
But here's the rub: Simkovic & McIntyre do not appear to have measured juris doctor holders.
The problem here is that the Census Bureau's Survey of Income and Program Participation ("SIPP"), used by Simkovic & McIntyre to build their sample set, does not appear to ask respondents whether they have a juris doctor. Instead, one of the SIPP topical modules (Wave 2, as viewed with the Census Bureau's DataFerret tool) asks respondents whether they have a "professional degree."
"Professional degree" does include the J.D as one example of a "professional degree," but there are two other categories that law graduates may possibly select outside of normal response error: master's and doctorate. What Simkovic & McIntyre appear to have done is cross-reference that answer with a separate question on the topical module asking what "field" the individual received that degree in.
In a footnote (fn 16 in this version, which may be a draft?) (see below), Simkovic & McIntyre explain that they required respondents to identify as having a "professional degree" in the field of "law." They elaborated that they excluded some unexplained number of people identifying as having a master's or doctorate or "imputed to be lawyers."
This approach may be both over-inclusive and under-inclusive in context. It's over-inclusive because there are other "professional" degrees in the field of law besides the juris doctor. Foreign-educated attorneys, for example, may rightly claim to have such degrees. Additionally, it is impossible to know if the sample is polluted with advanced paralegal degrees or others who have any of the myriad degrees that have spawned in the last twenty years.
It's under-inclusive because it's incredibly likely that some juris doctor holders checked a different box. Don't take my word for it; look at the actual numbers.
Looking at the 2008 SIPP responses and comparing people who selected "law" as their field of study in the Wave 2/topical module among various types of degrees identified, one sees the following:
Master's: 54
Professional School: 371
Doctorate: 67
By their explicit words, Simkovic & McIntyre apparently only included the 371 "professional" degree holders.
But what about the 67 people who identified having a "doctorate" in "law"? Are we assuming that none of these people have a juris doctor? Because I would imagine almost all of them do - it isn't like traditional PhD's in "law" are common (particularly because "social sciences" only has 33 and "liberal arts" only has 14 for the 2008 sample).
From a statistical purity standpoint, we have a big problem: people who almost certainly should be included are purposefully excluded by the study's apparent design. Not just 1 or 2%, but it's quite possible that 15% of the 2008 SIPP juris doctor holders were excluded as not having "law degrees."
And that's why the definition matters. When we ask law schools to collect data on their own graduates, we know those individuals have juris doctor degrees. If we exclude someone, it is because (in theory) we lack the data.
When Simkovic & McIntyre infer that a person has a "law degree" based on having a "professional" degree in "law" in census records, they introduce two layers of response error and use imprecise terminology.
The result is that the Economic Value may include people who don't have a juris doctor, almost certainly excludes people who do have a juris doctor, and tries to implicitly slide by on the premise that what they infer to be "law degrees" based on an unrelated governmental survey actually are people with juris doctor degrees.
Simkovic & McIntyre have repeatedly emphasized that other studies are over- or underinclusive and theirs hits the sweet spot:
Our data sources enable us to estimate earnings premiums and increased labor force participation attributable to a law degree, not only for the under-inclusive category of lawyers or the over-inclusive category of professional degree holders, but for the appropriate group, law degree holders.Economic Value, SSRN Version, p. 12.
When you consider that the tracking of lawyers includes only people who have a JD or the equivalent, or that standardized law school tracking already includes the exact, dead-on population, such a claim appears to be totally unsupportable bullshit in light of SIPP's limitations of definition.
That Simkovic & McIntyre - and numerous others - have persisted in using this study to purportedly support conclusions about the population of ABA-accredited juris doctor holders - even though the study's explanations of how it defined such a population appear to be unacceptably imprecise on their face - is a testament to Simkovic & McIntyre's slick genius.
Most scientists would never be able to pull off using a bad, imprecise data set as a superior means of study than using records sampled from the exact population. The most renowned scholars on Earth - Hawking, Dawkins, Andrew Fucking Wiles - would never dare pass off an imprecise definition as the one true authentic representation of a million-person population.
But Simkovic & McIntyre did. It may significantly alter the results of their (still-otherwise flawed) study; it may not have any effect at all. The fact that it exists without any real explanation shows that Simkovic & McIntyre clearly belong in the "genius" category regardless of how we define the term.
Note: there appear to be multiple versions of Economic Value circulating on the internet. If anyone is aware of a "final" version that addresses any issue raised in this post or any subsequent post, please comment or forward a link to my attention. In any event, the author expressly disclaims any purported factual allegations or implications in this or any other part of the series and states affirmatively that he/it is a moron of improper breeding and education and that readers should defer to the educated professorate to guide our thoughts.
Monday, July 4, 2016
The Genius of Simkovic & McIntyre (Part I)
Happy 4th. The LSTC is taking a sabbatical of sorts in July. In the meantime, please enjoy some re-cycled, but previously unpublished, posts with critical praise for Simkovic & McIntyre's Economic Value from the blog's draft archives with slight editing.
At some point in his career, prior even to attaining the age of thirty, long before his most immortal songs were recorded, and before Jamie Foxx was even born, Ray Charles came to have "Genius" as a nickname for his synthesis of what were then otherwise segregated musical styles. It stuck on some level beyond a transparent marketing gimmick and became reflected in numerous album titles up through the man's death over forty years later.
Because Michael Simkovic and Frank McIntyre have similarly brought a genius-like revolutionary synthesis of academic demagoguery and lousy statistics, it should be appropriate that their hits receive similar appellation.
Consider Economic Value. Somehow, this paper still manages to be cited with seriousness and belief that the conclusions made therein are valid, scientific, and even worthy of discussion.
But science and statistics do not follow the Rules of Evidence. There is either good science or bad science. Bad science does not become "admissible" because it looks like good science enough to sneak past a judge (or, in this case, a pool of cronies). Once you hit "invalid" or "bad," you can - and should - disregard the conclusions. Bad science is trash. A thousand bags of trash do not "contribute to the academic discourse" so much as they attract flies and maggots.
Simkovic & McIntyre's research is wrong. Use whatever word you please: invalid, erroneous, statistically-unsupported, flawed, etc. Wrong is the most accessible term. This doesn't necessarily mean their calculations are wrong or that their politics are wrong or that they did not copy data correctly, but it does mean that the research does not offer viable, usable conclusions and should be as disregarded as-if I were to write "x + y = hypodermic unicorns" or begin proclaiming new-Earth creationism as the one true scientific truth.
The people who promote Economic Value and its ilk want the evidence admissible and there to be a liberal arts and/or courtroom-style debate between polished experts. In such a manner, fatal flaws can be excused with attacks on the opposition and a reference to numbers and statistical concepts that unfortunately bewilder many lawyers and legal commentators like a Cro-Magnon man given a microwave oven.
All the persuasive writing courses in the world cannot make bad science into good science.
Good science follows established protocols. In the realm of statistics, there needs to be a well-defined population that's included or excluded from study. Ideally, any sampling from that population is random and representative. Whether a study is empirical like most social sciences research or prospective like a clinical trial, variables (the particular traits being studied) should be properly isolated using controls and accounting for potentially conflicting variables.
Although the practical application of statistics can be remarkably complicated, the basic concepts of statistics are fairly simple and intuitive - basically, we want to compare apples to apples and make sure that any conclusions we draw about apples are as sound as possible.
In Economic Value, Simkovic & McIntyre study an imprecise population with a too-small, non-representative sample compared to the wrong control group while ignoring other research/conflicting variables and alternative explanations for their results.
In other words, apples taste sweeter than orange golf balls because apple seeds contain trace amounts of cyanide.
Bad science is bad science. Nonetheless, Simkovic & McIntyre have not only managed to peddle bad science thinly veiling aims of academic politics, but have it published and repeated across the legal education industry based on little more than it appears to "contribute" to the scholarly "debate" about the value of the law school.
Any dork with Excel can make bad statistics. But to package them in such a way that otherwise intelligent-on-paper people look past fatal flaws and accept it as anything other than a research project that should've been dropped in the toilet with the morning deuce?
That's genius. In the coming posts, the LSTC will explore the beauty of this genius applied on various levels.
At some point in his career, prior even to attaining the age of thirty, long before his most immortal songs were recorded, and before Jamie Foxx was even born, Ray Charles came to have "Genius" as a nickname for his synthesis of what were then otherwise segregated musical styles. It stuck on some level beyond a transparent marketing gimmick and became reflected in numerous album titles up through the man's death over forty years later.
Because Michael Simkovic and Frank McIntyre have similarly brought a genius-like revolutionary synthesis of academic demagoguery and lousy statistics, it should be appropriate that their hits receive similar appellation.
Consider Economic Value. Somehow, this paper still manages to be cited with seriousness and belief that the conclusions made therein are valid, scientific, and even worthy of discussion.
But science and statistics do not follow the Rules of Evidence. There is either good science or bad science. Bad science does not become "admissible" because it looks like good science enough to sneak past a judge (or, in this case, a pool of cronies). Once you hit "invalid" or "bad," you can - and should - disregard the conclusions. Bad science is trash. A thousand bags of trash do not "contribute to the academic discourse" so much as they attract flies and maggots.
Simkovic & McIntyre's research is wrong. Use whatever word you please: invalid, erroneous, statistically-unsupported, flawed, etc. Wrong is the most accessible term. This doesn't necessarily mean their calculations are wrong or that their politics are wrong or that they did not copy data correctly, but it does mean that the research does not offer viable, usable conclusions and should be as disregarded as-if I were to write "x + y = hypodermic unicorns" or begin proclaiming new-Earth creationism as the one true scientific truth.
The people who promote Economic Value and its ilk want the evidence admissible and there to be a liberal arts and/or courtroom-style debate between polished experts. In such a manner, fatal flaws can be excused with attacks on the opposition and a reference to numbers and statistical concepts that unfortunately bewilder many lawyers and legal commentators like a Cro-Magnon man given a microwave oven.
All the persuasive writing courses in the world cannot make bad science into good science.
Good science follows established protocols. In the realm of statistics, there needs to be a well-defined population that's included or excluded from study. Ideally, any sampling from that population is random and representative. Whether a study is empirical like most social sciences research or prospective like a clinical trial, variables (the particular traits being studied) should be properly isolated using controls and accounting for potentially conflicting variables.
Although the practical application of statistics can be remarkably complicated, the basic concepts of statistics are fairly simple and intuitive - basically, we want to compare apples to apples and make sure that any conclusions we draw about apples are as sound as possible.
In Economic Value, Simkovic & McIntyre study an imprecise population with a too-small, non-representative sample compared to the wrong control group while ignoring other research/conflicting variables and alternative explanations for their results.
In other words, apples taste sweeter than orange golf balls because apple seeds contain trace amounts of cyanide.
Bad science is bad science. Nonetheless, Simkovic & McIntyre have not only managed to peddle bad science thinly veiling aims of academic politics, but have it published and repeated across the legal education industry based on little more than it appears to "contribute" to the scholarly "debate" about the value of the law school.
Any dork with Excel can make bad statistics. But to package them in such a way that otherwise intelligent-on-paper people look past fatal flaws and accept it as anything other than a research project that should've been dropped in the toilet with the morning deuce?
That's genius. In the coming posts, the LSTC will explore the beauty of this genius applied on various levels.
Subscribe to:
Posts (Atom)