It has typically been assumed that habitat destruction, characterized by habitat loss and fragmentation, has consistently negative effects on biodiversity. While numerous empirical studies have shown the detrimental effects of habitat loss, debate continues as to whether habitat fragmentation has universally negative effects. To explore the effects of habitat fragmentation, we developed a simple model for site‐occupancy dynamics in fragmented landscapes. With the model, we demonstrate that a competition–colonization trade‐off can result in nonlinear oscillatory responses in biodiversity to both habitat loss and fragmentation. However, the overall pattern of habitat loss reducing species richness is still established, in line with empirical observations. Interestingly, the existence of localized oscillations in biodiversity can explain the mixed responses of species richness to habitat fragmentation per se observed in nature, thereby reconciling the debate on the fragmentation–diversity relationship. Therefore, this study offers a parsimonious mechanistic explanation for empirically observed biodiversity patterns in response to habitat destruction.