Summary In many applications of two‐component mixture models such as the popular zero‐inflated model for discrete‐valued data, it is customary for the data analyst to evaluate the inherent heterogeneity in view of observed data. To this end, the score test, acclaimed for its simplicity, is routinely performed. It has long been recognised that this test may behave erratically under model misspecification, but the implications of this behaviour remain poorly understood for popular two‐component mixture models. For the special case of zero‐inflated count models, we use data simulations and theoretical arguments to evaluate this behaviour and discuss its implications in settings where the working model is restrictive with regard to the true data‐generating mechanism. We enrich this discussion with an analysis of count data in HIV research, where a one‐component model is shown to fit the data reasonably well despite apparent extra zeros. These results suggest that a rejection of homogeneity does not imply that the underlying mixture model is appropriate. Rather, such a rejection simply implies that the mixture model should be carefully interpreted in the light of potential model misspecifications, and further evaluated against other competing models.