I'm sure this posting will generate a similar response among readers as the earlier post on discriminatory NBA refs.
But I wonder if anyone shares my concern-- How do economistst get themselves into a position where they are willing to put their reputations on the line for a given working paper before it has passed critical muster?
I had a similar inquiry from SI.com last month about a working paper on discrimination in firing of NBA coaches. I firmly refused to allow the reporter to go there since it's not in print.
Comments?
Hi Rod,
I guess you are about settled after your move. I hope it went well.
I am not sure I understand your query. If your concern is that people are commenting on a paper they have not read as if they know it well, then I fully agree.
If instead it is that people are discussing the paper at all without it having been refereed and published, then I don't think I agree with your concern. Discussing the work before publication may help the authors to make it better, in the same way that discussants at conferences are supposed to perform that function. The discussion may stimulate collaboration on related ideas or topics. That, it strikes me, is the whole point of working papers.
Of course, I don't like the idea that working papers get played up in the popular press before they have been sufficiently vetted within the profession. If that is your concern, I share it completely. It seems to me that we run the risk of looking foolish as individuals and as a profession - a bit like those fellows that reported a break through in nuclear fusion a few years ago, only to be shown to be wrong shortly thereafter.
But then again, in the environment where bringing renown to one's institution is important for tenure, promotion, and salary increases, the incentive to get one's name before the public is perhaps strong. So what is the alternative for some assistant professor needing a bit of a boost to their chances of promotion and tenure but to go for some publicity?
I sat on the paper for ~2 weeks while on vacation. Once the media (Science News, MSNBC, Time, etc) pick up the paper, I see no point in holding back on posting of its existence. The horse has left the barn.
The nuclear fusion example on pre-publication is a good one, but overstated. Getting the thing published is no guarantee that the finding will not be overturned. If a senior chaired professor like Dan Hamermesh is willing to talk about his findings and take that risk prior to publication, that's fine with me.
I found the discussion among sports economists following the NBA paper - indeed the entire reaction to it, led by the NBA - to be over the top. But Wolfers was going around the country presenting the paper at conferences, so I don't see why he should be banished from the guild for talking to a New York Times reporter.
I obviously disagree with Rod on this point. I also probably violated his maxim when reporters called to talk about my "Moneyball" paper with Jahn Hakes. That paper was in the top 5 of SSRN downloads at the time, and there was interest in it. I'd have been foolish to have kept my mouth shut while the AER sat on the paper for a year, before rejecting it. If it weren't for the entrepreneurial approach taken by the editors at JEP, that paper would probably still be unpublished and I'd still be mum.
There's another subtle issue here: Increasingly we don't get to choose when a paper "goes public". There is enough interest among print journalists in economics topics these days that they are - literally - trawling conferences and websites looking for interesting research to write about. In the case of my NBA refs study with Joe Price, the press managed to get hold of it before we had ever even posted it on the web. (Someone at a seminar distributed a copy to a journalist, despite the warning "Not for circulation" on the cover page.)
Once one's research is going to be discussed publicly, then I do think we have a responsibility to do what we can to make sure the subsequent discussion is informed and sensible. (And I'll admit that we didn't fully succeed on the NBA refs study.)
Here is my question and it has nothing to do with when a paper is released. I haven't read the paper yet. Being a huge Orioles fan and having to deal with Daniel Cabrerra as starting pitcher, the announcers always make the comment about how pitchers reputations are taken into consideration. Does the paper take this possibility into consideration?
This brings up the question of whether or not race makes a different when it comes to reputation?
My comment wasn't about discussion among economics professionals. Nothing should hinder that.
And it sounds as if I'm in the minority on the issue of "going public."
I try to do the best I can, but then discover through the presentation/review process that what I thought was great work just wasn't.
While the presentation/review process isn't perfect, it's pretty good.
Anyway, we're all free to choose, but I'll choose to tell reporters that my unpublished work hasn't passed muster yet and leave it at that.
Thank you all for your input.
I started ready the paper. One thing that I am not sure that they measure is where the umpire sets up. I am sure if this information is even kept by anyone. For example, if the umpire sets up inside, he is less likely to be able to see the outside of the strike zone. This could impact the bias on what is a strike and with the consistency.
Matt, My guess is that you are right on both points -- reputation & positioning. The advantage of questec technology is that it can be used to monitor & improve performance of the umps. If baseball cares, they can create the measures of your variables and do that analysis. I doubt that would overturn the findings of this paper though.
Rod,
Do you disagree with Justin's point then, that if one's work is already the subject of public discussion despite attempts to avoid its public release we should participate to try to keep the discussion informed and sensible?
Everyone,
Matt suggested that where the umpire sets up could matter for the bias. Skip suggests he doubts that control for this would affect the bias results. I tend to agree with Skip's reaction. My question is whether where umpires set up has enough variation to account for anything. Do umpires set up differently across pitches to the same batter or do they always set up on the outside, or inside, regardless of the hitter? Are there league guidelines about where the umpire sets up? I remember watching a game years ago, Tony Kubek may have been in the booth with Joe Garagiola, in which they commented on different strike zones in the American and National Leagues because in one league the chest protector was worn on the inside and in the other it was worn on the outside.
Hi Dennis.
No,I don't disagree with Justin. I'm just saying that if a reporter is going to do a story on MY work in progress, I'll request that they don't.
If they persist, I'll insist-- Quote me as saying I would rather this story did not appear. If they still persist, then so be it. Some things are simply out of my control. But at least I tried to make the better outcome (IMHO) occur.
By the way, given the notched-up fact checking happening lately, it's probably the case that the reporter would back off anyway.
I think the topic and conclusions of the research play a role in determining the author's obligation for care and prior review. In this case, the bar should be set very high: the claim that "umpires are racist" is newsworthy and gets lots of attention, while the contrary finding would be ignored. Unlike a paper on, say, the economic impact of subsidized stadiums, there are no experts available to reporters to refute the finding (unless MLB tries to, which they've wisely refrained from). And even if a definitive refutation of this paper comes along later, it will be little noticed (outside academia).
As a result, fans now "know" that umps are racially biased, and it will be very hard to change that perception. Researchers have a duty to be extremely careful with a topic like this one, because you can't unscramble this egg.
That's quite unfortunate, because the authors' conclusions are significantly overstated, if not wrong. They (and the media), describe the impact as "about 1 pitch per game." In fact, it is about one pitch per 5 games: having a same-race ump increases the called strike rate by .0034, and there are about 70 called pitches per game.
Look at table 2 in the paper if you want to see the magnitude of the bias we're talking about. White pitchers get a strike call .3206 from white umps, and .3192 from minority umps (a difference of 1 pitch in 700). If you re-run the numbers assuming that Hisp pitchers faced same-race umps 91% of the time (as white pitchers now do), their called strike rate would increase .0029. That means 4.6 pitches over a 200 inning season, or about one-third of a run. For an average pitcher, his ERA would drop from 4.20 to 4.19! Moreover, the change for black pitchers facing 91% black umps is only half this great. Or turn it around and assume white pitchers faced opposite-race umps 95% of the time, as minority pitchers do today: again, the impact is less than half of the Hispanic-Hispanic example.
There are good reasons to doubt whether this data in fact reveals any discrimination at all. But these issues have been addressed elsewhere. My point here is that, even if you completely accept the authors' finding, their interpretation has been exaggerated and somewhat irresponsible, given the extreme sensitivity of their topic. Ninety three professional umpires have potentially been libeled, and it's not clear who will clear their name, or how.
Guy,
You claim the finding is trivial, and accuse a fine person, who I am certain has no axe to grind on this issue, of libel. Please examine closely the right panel of figure 1 in the paper.
Can you can explain that result to me - minority umps calling about 1.6% more strikes for minority pitchers in non-questec parks? That is where the differences -- admittedly small -- come from.
Skip, the result you cite is not statistically significant. It's only 1.8 SD.
Hi all.
I almost hate to chime in again, but the last three posts are precisely why I take the "no comments for the press on unpublished journal work" stance.
Academic grab ass is one thing. Public consumption is quite another.
I basically duplicated this study using Retrosheet pitch data, which should be the same as the data used by these authors. I did not use a regression analysis though, so my data should be much easier to follow and "believe" assuming that the intermediate data and the adjustments are done correctly.
My final results are in posts 24 and 25 at this we address:
http://www.insidethebook.com/ee/...ussion_i_think/
Skip
Also the right panel of figure 1 (Questec/ non-Questec) probably shows a strong park bias. Questec is only in certain parks and is certainly not randomly distributed as some teams (home teams in particular) will be over-represented.
Skip: First, I said the umps had "potentially" been libeled -- a rather more careful statement than the conclusions made by the authors. You say that one of the authors is a "fine person," which I have no reason to doubt. But if it happened that you had a friend who was a white MLB ump, but knew no labor economists, do you think your reaction might be a little different?
As Phil and John note, your specific example is not strong evidence of bias. The samples of same-race matchups in Questec and non-questec parks are both small and very non-random. It's also interesting that if your example does indicate bias, it actually suggests bias by MINORITY umps. More generally, their data is consistent with a finding of bias mainly or only by minority umps, yet the authors entirely ignore this and actually conclude that "In particular, non-White pitchers are at a significant disadvantage relative to their White peers."
Are you not troubled by the fact the authors exaggerate the impact of the bias by a factor of 5? Do you really think that a difference that increases ERA by .01 runs is significant in any way, and worthy of an article in Time? Moreover, since this level of bias (1 per 700 pitches) couldn't possibly be discerned and exploited by pitchers or hitters, the authors then have no plausible theory for how umpire race could impact HRs and other offensive events. Until/unless someone does, any differences they find in those areas should be excluded as likely random fluctuations (and their failure to control for park, league, and pitcher race in those regressions makes the findings unconvincing).
You also say the author has "no axe to grind." This isn't true, of course. Hamermesh studies discrimination, and expected to find it here. After all, he told Time "We all have these subconscious preferences for our own group." Someone who expects to find bias may keep searching if it doesn't immediately reveal itself, but stop too quickly if he thinks he sees is. Moreover, a finding of discrimination makes the paper much more consequential, both in academia and publicly, while a finding of no discrimination would have been ignored. So he potentially brings at least two relevant biases to the table. Neither of these mean we should reject his findings out of hand -- all researchers have some potential bias of this kind. But it means that he (and we) have a heightened obligation to scrutizine the analysis closely.
Perhaps there is some very small amount of bias. My point is only that this evidence is far from conclusive, and in a situation like this where the indictment amounts to a (nearly irreversible) conviction, researchers should not go public until the work has been thoroughly reviewed and vetted. Even better is to allow for review by a community larger than fellow economists, who sometimes may -- as you have nicely illustrated here -- provide knee-jerk and uncritical support to their colleagues (as all profe
Rodney, why do those 'last three posts' to which you refer make you believe the study should be not discussed in public? I'm not sure if you mean (a) flawed papers should not be discussed publicly until academic critics have had at it, or (b) the kind of discussion we're having here is not of benefit.
I think it's (a), but I'm not sure. 
Oops... my last post should conclude "...provide knee-jerk and uncritical support to their colleagues (as all professions are wont to do)."
There is a good discussion at The Book blog site as well. Also, I need to redo my batter and pitcher adjustments. Guy, excellent post above. Your use of the words "potentially libel" ("potentially" being a key word) was perfectly appropriate given the context.
Also, I wrote to one of the authors (Hamermesh) as I only had 2 Hispanic umps and 4 black umps and A LOT fewer black pitchers in my research. I asked him if he could send me a list of his pitcher and umpire breakdowns. He sent a nice response, but oddly, he said he had to contact the other two authors. I thought that was a little strange. I can't imagine that the list of pitcher and umpire categories can be propietary. Quite the contrary, in fact. All of the data in a study MUST be avavilable and transparant, right? Does anyone know who the 5th black umpire and 3rd Hispanic one is? I had Diaz, Meriwether, Danley, and Bucknor as black, and Hernandez and Marquez as Hispanic. The ONLY black pitchers I had were Baldwin, Gordon, Hawkins, King, Oliver, Rhodes, Sabathia, Willis, and Jerome Williams. The authors had 27 black pitchers!
OK, I retract what I said about it being "odd" that Dr Hamermesh did not send me the data. I misread his email. He simply said that he would forward it to his colleagues who collected the data. My apologies to Dr. Hamermesh.
Hi Phil.
I meant a version of (a). Earlier in this line I already addressed (b); nothing should stop reasonable interested discussion.
My version of (a) needs more specific definitions-- "public" means reporters in the popular media. I would never exclude the type of "public" discussion/critical analysis at this blog, for example.
Sorry if I wasn't clearer.
I think that the authors may have to consider another relatively insidious variable:
Game time temperature.
There is an axiom among umpires that "strikes get you outs and outs get you home". This would almost certainly be more prevalent (and bubble into conciousness) on particularly hot or cold days. These days (in non QuesTec parks) would also likely have lower attendance and the cost of this expediency would also be lower.
I would not assert that this effect is greater than the race effect, which is laid out very well, but it could be another key variable.
Am I the only one who thinks the bottom of the argument is that 1% is not statistically significant AT ALL? For Hammermesh to suggest that his finding is signisficant is absurd, and obviously suggests that he did, in fact, go into his study looking to find bias, and when he found that there was so little, decided to press on, anyway.
"I expect that [MLB] will not be very happy about this, but the fact that with a little bit of effort this kind of behavior can be altered, that's very gratifying. I wish with society as a whole we could reduce the impact of discrimination as easily as it could be done in baseball."
Is he kidding somebody? What kind of behavior? The kind of behavior that you need a microscope to see? Headlines have followed Hammermesh, and that's all. His study proves nothing.
I am not an economist or a statistician, but I wonder if there are problems with the racial/ethnic classification of umps and players. "Hispanic" is not a race; but rather an ethnic group. Hispanics can be to all appearances white, black, Indian (native) or Asian, but are usually some mixture. Often, ethnicity and race are asked separately, so a Hispanic person would fill in a separate bubble for race. Perhaps these definitional issues are accounted for in the statistical studies (I'm sure this comes up all the time) and I am raising a red herring, I just wanted to throw it out into the discussion.
Another issue I have is with the way Hispanics are defined: "Players or umpires are classified as Hispanic if they are born in one of the following countries: Colombia, Cuba, Curacao, Dominican Republic, Mexico, Nicaragua, Panama, Puerto Rico or Venezuela." So a person of Hispanic roots born in the U.S. is not Hispanic? That seems odd, especially considering the number of Hispanics born in the United States.
I agree with John Perricone's point above that the slim difference in calls do not demonstrate bias to a significant degree at all.
I wonder also, and I have not seen it addressed in my first buzz through the material, if given the razor thin difference of 1% in aggregate calls, is there a possibility that if the data were parsed out by years 2004, 2005 & 2006 instead of aggregated, might we see no bias or reverse bias in one or two years and bias in the other years?
And then what does that say except that there might be no bias at all, but a coin flip chance of seeing bias (Yes/No)?
Commenting by HaloScan
