In situations in which one seels simply to evaluate the cumulative significance of a number of tests of a particular hypothesis, it is well known that people will seriously overestimate the negative evidentiary weight that should be attached to failures to produce significant replications of previously significant findings and will therefore be likely to underestimate the overall signifcance of small but consistence effects. Meta-analysis may help to provide an answer to this problem. In situations in which one seeks to examine more complex and theoretivally-driven literatures, however, other problems appear. It is well-documented that people tend to interpret and construe research evidence in a biased fashion, taking at face value results that support views while subjecting to intense critical scrutiny findings that contradict those beliefs. Reviewers and journal editors, similarly, are certainly guilty of favouring `positive` over `negative` results. Meta-analysis has also been offered as an answer to these problems. The crucial question in the latter case is whether the proposed `cure` of meta-analysis may not be worse than the `disease` of potentially biased interpretation. It is not that meta-analysis will necessarily lead to inappropriate conclusions. Instead the point is that the putative `objectivity` of meta-analysis actually seems no greater guarantee than traditional methods that a literature review will yield conclusions that will stand the test of time.
Abstract