In this work, we propose a system to track the clouds and predict relevant events based on all-sky images. To deal with the nature of variable appearance of clouds, we use clusters of feature points to perform tracking. We propose an enhanced clustering algorithm that does not require prior knowledge of number of clusters. The proposed clustering algorithm can successfully separate feature points into groups with reasonable sizes and ranges. In the tracking process, merging and splitting of clouds are handled via checking matched pairs of feature points among different clusters. Afterwards, the tracking information is utilized to predict if the sun will be covered or obscured by clouds within the prediction horizon. Features are extracted from the tracked feature points and a Markov chain model is designed to perform ramp-down event prediction. The obscuration and ramp-down events have an important impact on solar irradiance. The experiments have shown that the proposed system can substantially enhance the accuracy of solar irradiance nowcasting on a challenging dataset. •Cloud tracking is achieved using clusters of feature points.•An enhanced clustering algorithm is proposed to prevent over-clustering or under-clustering.•Obscuration events are predicted with features of clustering and tracking information.•A graph based model is designed to prediction ramp-down events.•Detection results are applied in irradiance nowcasting frameworks to improve accuracy.