How Many Citations Are Fawlty in the Average Economics Paper?

My advisor recently sent me a terrific paper - The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?. A PDF of the full paper is available here. Some key points from the abstract:

Furthermore, even when the paper was cited, 49 of the 50 studies that we examined reported its findings improperly. The inappropriate use of statistical-significance testing led researchers to conclude that nonresponse bias was not present in 76 percent of the studies in our sample. Only one of the studies in the sample made any adjustment for it. Judging from the original paper, we estimate that the study researchers should have predicted nonresponse bias and adjusted for 148 variables. In this case, the faulty citations seem to have arisen either because the authors did not read the original paper or because they did not fully understand its implications.

The recommendations of the authors are good, but they fail to address the heart of the problem:

To address the problem of omissions, we recommend that journals include a section on their websites to list all relevant papers that have been overlooked and show how the omitted paper relates to the published paper. In general, authors should routinely verify the accuracy of their sources by reading the cited papers. For substantive findings, they should attempt to contact the authors for confirmation or clarification of the results and methods. This would also provide them with the opportunity to enquire about other relevant references. Journal editors should require that authors sign statements that they have read the cited papers and, when appropriate, have attempted to verify the citations.

The problem is clearly one of incentives. Journal articles are refereed, but referees have a rather peculiar set of incentives. I referee a handful of papers on my topic a year and I face the following incentives:

• I get paid the same (nothing) whether I spend a lot of time on the paper or a little, or whether I provide a useful report or a useless one.



• Citations are particularly important for me in obtaining tenure. So I will want to get my papers cited wherever possible.



• The authors of the paper being submitted are competing for the set small set of jobs that I am.

If I am at all rational, I should not spend a lot of time on the referee report, I should use the report as a mechanism to promote my own work and I should not go out of my way to be helpful. This pretty much describes every referee report I have ever received as an author. My all-time favorite was a report that blasted me for not reporting the findings of some "key" paper in my field. I was worried that I might have missed an important work on my topic (that may have made my paper redundant). Until I discovered that the "key" paper:

• Was never published and was only available as a working paper.



• Was only available on the website of the author, a Ph.D. student.



• Was not even on the same topic as my paper.

I think that was the day I would be more valuable to the private sector than in academia.

If economists believe at all that academics are rational, then I do not know how they expect the review process, as it currently stands, to provide anything of value. Incentives matter!