•We build a dataset, which includes video and time-sync comments, for segment popularity prediction.•We study the problem of video segment popularity prediction, which can help audiences to find attractive shots.•With the help of time-sync comments information mining, the model can better understanding the audiences.•Combining visual and text information can make results more precise. With the rapid development of digital equipment and the continuous upgrading of online media, a growing number of people are willing to post videos on the web to share their daily lives (Jelodar, Paulius, & Sun, 2019). Generally, not all video segments are popular with audiences, some of which may be boring. If we can predict which segment in a newly generated video stream would be popular, the audiences can only enjoy this segment rather than watch the whole video to find the funny point. And if we can predict the emotions that the audiences would induce when they watch a video, this must be helpful for video analysis and for guiding the video-makers to improve their videos. In recent years, crowd-sourced time-sync video comments have emerged worldwide, supporting further research on temporal video labeling. In this paper, we propose a novel framework to achieve the following goal: Predicting which segment in a newly generated video stream (hasn’t been commented with the time-sync comments) will be popular among the audiences. At last, experimental results on real-world data demonstrate the effectiveness of the proposed framework and justify the idea of predicting the popularities of segments in a video exploiting crowd-sourced time-sync comments as a bridge to analyze videos.