A large number of studies have presented empirical arguments for the existence of alternative stable states (ASS) in a wide range of ecological systems. However, most of these studies have used non-manipulative, indirect methods, which findings remain open for alternative explanations. Here, we review the direct evidence for ASS resulting from manipulation experiments. We distinguish four conclusive experimental approaches which test for predictions made by the hysteresis effect: (1) discontinuity in the response to an environmental driving parameter, (2) lack of recovery potential after a perturbation, (3) divergence due to different initial conditions and (4) random divergence. Based on an extensive literature search we found 35 corresponding experiments. We assessed the ecological stability of the reported contrasting states using the minimum turnover of individuals in terms of life span and classified the studies according to 4 categories: (1) experimental system, (2) habitat type, (3) involved organisms and (4) theoretical framework. 13 experiments have directly demonstrated the existence of alternative stable states while 8 showed the absence of ASS in other cases. 14 experiments did not fulfil the requirements of a conclusive test, mostly because they applied a too short time scale. We found a bias towards laboratory experiments compared to field experiments in demonstrating bistability. There was no clear pattern of the distribution of ASS over categories. The absence of ASS in 38% of the tested systems indicates that ASS are just one possibility of how ecological systems can behave. The relevance of the concept of ASS for natural systems is discussed, in particular under consideration of the observed laboratory bias, perturbation frequency and variable environments. It is argued, that even for a permanently transient system, alternative attractors may still be of relevance.