The replication crisis in the social and psychological sciences is said to be due in part to publication bias and the resulting file drawer problem. Meta-analysis is often advocated as a means of resolving this crisis but is prone to the same publication bias and file drawer effects. A study of 23 meta-analyses examined the consequences of correcting for these effects through the inclusion of unpublished research. Results indicated that the inclusion of unpublished data led to smaller meta-analytic means in some instances, consistent with the primary theorized effects of the file drawer problem and publication bias, but led to larger meta-analytic means in nearly as many instances, suggesting the presence and influence of other, unknown artifacts in unpublished research. These findings suggest that meta-analysis is limited in its ability to resolve the replication crisis and instead may introduce unexplained effects that render cumulative science problematic.