Display omitted •The niche of dominant species is an intuitive embodiment of water nutrient pattern.•Niche overlap index quantifies the boundaries between dominant species.•The effect of N: P accelerated the diffusion rate of Pseudoanabaena limnetica.•Application of niche-based theories and neutral-based theories to phytoplankton.•Results facilitate early warning of cyanobacteria blooms in karstic reservoirs. In aquatic ecosystems, the dynamics of ecological parameters play an important role in community organization. To understand the changes in the phytoplankton community structure in karstic reservoirs, the Hongfeng reservoir in Guizhou plateau was monitored monthly from 2016 to 2019. The results showed that the dominant organisms in phytoplankton comprised five phyla and twelve species during 4 years, among which the proportion of abundance of Pseudoanabaena limnetica begin to rise in 2018 and reached 46.57% in 2019 (the highest during the study period). The niche breadth (0.259–0.925) and overlap value (0–0.991) of P. limnetica have an obvious seasonal trend and are significantly affected by environmental factors such as water temperature, nutrients, and interspecific competition. This is primarily reflected in the high adaptability of the dominant species to changes in the N:P concentration ratio in water. Variance Partitioning Analysis (VPN) results show that the diffusion degree of P. limnetica is the result of the interaction between internal (i.e. interspecific competition, with a contribution rate of 13.4%) and external (i.e. environmental factors, with a contribution rate of 86.6%). The greater the external influence, the higher the diffusion degree. The results of niche-based theories and neutral-based theories showed that the stable community structure (variance rate (VR > 1)) and the excessive fast diffusion rate of dominant species (migration rate (m = 0.84)) may lead to the risk of P. limnetica and other toxic filamentous cyanobacteria blooms. The results provide basic data for the study of early warnings regarding phytoplankton community succession and cyanobacteria blooms in large reservoirs in the karst areas.