ABSTRACT Can new cosmic physics be uncovered through tensions amongst data sets? Tensions in parameter determinations amongst different types of cosmological observation, especially the ‘Hubble tension’ between probes of the expansion rate, have been invoked as possible indicators of new physics, requiring extension of the ΛCDM paradigm to resolve. Within a fully Bayesian framework, we show that the standard tension metric gives only part of the updating of model probabilities, supplying a data co-dependence term that must be combined with the Bayes factors of individual data sets. This shows that, on its own, a reduction of data set tension under an extension to ΛCDM is insufficient to demonstrate that the extended model is favoured. Any analysis that claims evidence for new physics solely on the basis of alleviating data set tensions should be considered incomplete and suspect. We describe the implications of our results for the interpretation of the Hubble tension.