To improve energy efficiency and maintain the stability of the power grid, time-of-use (TOU) electricity tariffs have been widely used around the world, which bring both opportunities and challenges to the energy-efficient scheduling problems. Single machine scheduling problems under TOU electricity tariffs are of great significance both in theory and practice. Although methods based on discrete-time or continuous-time models have been put forward for addressing this problem, they are deficient in solution quality or time complexity, especially when dealing with large-size instances. To address large-scale problems more efficiently, a new greedy insertion heuristic algorithm with a multi-stage filtering mechanism including coarse granularity and fine granularity filtering is developed in this paper. Based on the concentration and diffusion strategy, the algorithm can quickly filter out many impossible positions in the coarse granularity filtering stage, and then, each job can find its optimal position in a relatively large space in the fine granularity filtering stage. To show the effectiveness and computational process of the proposed algorithm, a real case study is provided. Furthermore, two sets of contrast experiments are conducted, aiming to demonstrate the good application of the algorithm. The experiments indicate that the small-size instances can be solved within 0.02 s using our algorithm, and the accuracy is further improved. For the large-size instances, the computation speed of our algorithm is improved greatly compared with the classic greedy insertion heuristic algorithm.