Matched molecular pair analysis (MMPA) has become a major tool for analyzing large chemistry data sets for promising chemical transformations. However, the dependence of MMPA predictions on data constraints such as the number of pairs involved, experimental uncertainty, source of the experiments, and variability of the true physical effect has not yet been described. In this contribution the statistical basics for judging MMPA are analyzed. We illustrate the connection between overall MMPA statistics and individual pairs with a detailed comparison of average CHEMBL hERG MMPA results versus pairs with extreme transformation effects. Comparing the CHEMBL results to Novartis data, we find that significant transformation effects agree very well if the experimental uncertainty is considered. This indicates that caution must be exercised for predictions from insignificant MMPAs, yet highlights the robustness of statistically validated MMPA and shows that MMPA on public databases can yield results that are very useful for medicinal chemistry.