Abstract This study investigates the Altmetric Attention Score (AAS) fluctuations and altmetrics source stability in Information Science & Library Sciences publications, analyzing 26,474 documents from 2012 to 2021. It finds that 23.7% of these publications experienced AAS fluctuations over a year. Significant influences on AAS variability include Twitter mentions and policy documents, with Twitter mentions showing high volatility, affecting nearly 30% of papers. The removal of the Analysis & Policy Observatory as a policy source led to a notable drop in policy document mentions. Various types of mention vanishing are identified, such as administrative decisions by altmetrics aggregators, technological issues, user actions on digital platforms, and natural changes on platforms like Wikipedia. In response, the study proposes modifications in the tracking and monitoring system of mentions to not overlook these vanishings. This aims to enhance the reliability and stability of altmetrics at a time when there is a call for their use in the research evaluation.