•We analyze the insights underlying the design of the disruption index.•Conceptual work is more likely to disrupt science than technical work.•Disruption reflects how paradigms shift in the development of science.•“New direction” identified by disruption also includes using new framing for an existing phenomenon that facilitates scientific dissemination.•Citing highly cited papers may decrease the disruption of paper. The disruption (D) index is a network-based indicator to quantify the extent to which a focal paper disrupts its predecessors. This study focuses on what disruption means by examining example articles related to “sleeping beauties in science” and frequency-inverse document frequency (TF-IDF). We investigated the structure of the citation network and subsequent papers’ motivations for citing the focal papers. Based on the observation that conceptual work is more likely to disrupt science than technical work, we hypothesize that disruption reflects the mechanism of how paradigms shift in the development of science. We also assume that the disruption identified by the D index indicates more than generating a new direction. Disruptive contributions include revolutionary studies such as Nobel-prize-winning papers, as suggested in previous work. However, disruptive contributions also include scientific dissemination of new terminology created by popular proposals, such as “sleeping beauties in science.” Such contributions redefine and popularize phenomena in science.