Quantifying the shape and strength of mating preferences is a vital component of the study of sexual selection and reproductive isolation, but the influence of experimental design on these estimates is unclear. Mating preferences may be tested using either no-choice or choice designs, and these tests may result in different estimates of preference strength. However, previous studies testing for this difference have given mixed results. To quantify the difference in the strength of mating preferences obtained using the 2 designs, we performed a meta-analysis of 38 studies on 40 species in which both experimental designs were used to test for preferences in a single species/trait/sex combination. We found that mating preferences were significantly stronger when tested using a choice design compared with a no-choice design. We suggest that this difference is due to the increased cost of rejecting partners in no-choice tests; if individuals perceive they are unlikely to remate in a no-choice situation they will be more likely to mate randomly. Importantly the use of choice tests in species in which mates are primarily encountered sequentially in the wild may lead to mating preferences being significantly overestimated. Furthermore, this pattern was seen for female mate choice but not for male mate choice, and for intraspecific choice but not for interspecies or interpopulation mate discrimination. Our study thus highlights the fact that the strength of mating preferences, and thus sexual selection, can vary significantly between experimental designs and across different social and ecological contexts.Lay Summary Mating preferences in animals are stronger when individuals have more than one option to choose from. We performed a meta-analysis of studies in which mating preferences were tested using 2 experimental designs: no-choice tests and choice tests. This difference in preferences between designs was only seen for female, intraspecific mate choice. Individuals may be less choosy in no-choice tests because the likelihood of encountering another mate is perceived to be lower than in choice tests.