Biological invasions are both a pressing environmental challenge and an opportunity to investigate fundamental ecological processes, such as the role of top predators in regulating biodiversity and food-web structure. In whole-ecosystem manipulations of small Caribbean islands on which brown anole lizards (Anolis sagrei) were the native top predator, we experimentally staged invasions by competitors (green anoles, Anolis smaragdinus) and/or new top predators (curly-tailed lizards, Leiocephalus carinatus). We show that curly-tailed lizards destabilized the coexistence of competing prey species, contrary to the classic idea of keystone predation. Fear-driven avoidance of predators collapsed the spatial and dietary niche structure that otherwise stabilized coexistence, which intensified interspecific competition within predator-free refuges and contributed to the extinction of green-anole populations on two islands. Moreover, whereas adding either green anoles or curly-tailed lizards lengthened food chains on the islands, adding both species reversed this effect-in part because the apex predators were trophic omnivores. Our results underscore the importance of top-down control in ecological communities, but show that its outcomes depend on prey behaviour, spatial structure, and omnivory. Diversity-enhancing effects of top predators cannot be assumed, and non-consumptive effects of predation risk may be a widespread constraint on species coexistence.