Ecological drift causes species abundances to fluctuate randomly, lowering diversity within communities and increasing differences among otherwise equivalent communities. Despite broad interest in ecological drift, ecologists have little experimental evidence of its consequences in nature, where competitive forces modulate species abundances. We manipulated drift by imposing 40-fold variation in the size of experimentally assembled annual plant communities and holding their edge-to-interior ratios comparable. Drift over three generations was greater than predicted by neutral models, causing high extinction rates and fast divergence in composition among smaller communities. Competitive asymmetries drove populations of most species to small enough sizes that demographic stochasticity could markedly influence dynamics, increasing the importance of drift in communities. The strong effects of drift occurred despite stabilizing niche differences, which cause species to have greater population growth rates when at low local abundance. Overall, the importance of ecological drift appears greater in non-neutral communities than previously recognized, and varies with community size and the type and strength of density dependence.