Recent years have seen growing interests in developing and applying computer vision technologies to solve safety problems in the construction industry. Despite the technological advancements, there is no research that exams the theoretical links between computer vision technology and safety science and management. Thus, the objectives of this paper are to: (1) investigate the current status of applying computer vision technology to construction safety, (2) examine the links between computer vision applications and key research themes of construction safety, (3) discuss the theoretical challenges of applying computer vision to construction safety, and (4) recommend future research directions. A five-step review approach was adopted to search and analyze peer-reviewed academic journal articles. A three-level computer vision development framework was proposed to categorized computer vision applications in the construction industry. The links between computer vision and three main safety research traditions: safety management system, behavior-based safety program, and safety culture, were discussed. The results suggest that the majority of past efforts were focused on object recognition, object tracking, and action recognition, with limited research focused on recognizing unsafe behavior. There are even fewer studies aimed at developing vision-based safety assessment and prediction systems. Based on the review findings, four future research directions are suggested: (1) develop and test a behavioral-cues-based safety climate measure, (2) develop safety behavior datasets, (3) develop a formal hazard identification and assessment model, and (4) develop criteria to evaluate the real impacts of vision-based technologies on safety performance.