The β‐null deviation measure, developed as a null model for β‐diversity, is increasingly used in empirical studies to detect the underlying structuring mechanisms in communities (e.g. niche versus neutral and stochastic versus deterministic). Despite growing use, the ecological interpretation of the presence/absence and abundance‐based versions of the β‐null diversity measure have not been tested against communities with known assembly mechanisms, and thus have not been validated as an appropriate tool for inferring assembly mechanisms. Using a mechanistic model with known assembly mechanisms, we simulated replicate metacommunities and examined β‐null deviation values 1) across a gradient of niche (species‐sorting) to neutrally structured metacommunities, 2) through time, and 3) we compared the effect of changes in assembly mechanism on the performance of the β‐null deviation measures. The impact of stochasticity on assembly outcomes was also considered. β‐null deviation measures proved to be interpretable as a measure of niche or neutral assembly. However, the presence/absence version of the β‐null deviation measure could not differentiate between niche and neutral metacommunities if demographic stochasticity were present. The abundance‐based β‐null deviation measure was successful in distinguishing between niche and neutral metacommunities and was robust to the presence of stochasticity, changes through time, and changing assembly mechanisms. However, we suggest that it is not robust to changing abundance evenness distributions or sampling of communities, and so its interpretation still requires some care. We encourage the testing of the assumptions behind null models for ecology and care in their application.