In construction, about 80%–90% of accidents are associated with workers' unsafe acts. Nevertheless, the measurement of workers' behavior has not been actively applied in practice, due to the difficulties in observing workers on jobsites. In an effort to provide a robust and automated means for worker observation, this paper proposes a framework of vision-based unsafe action detection for behavior monitoring. The framework consists of (1) the identification of critical unsafe behavior, (2) the collection of relevant motion templates and site videos, (3) the 3D skeleton extraction from the videos, and (4) the detection of unsafe actions using the motion templates and skeleton models. For a proof of concept, experimental studies areundertaken to detect unsafe actions during ladder climbing (i.e., reaching far to a side) in motion datasets extracted from videos. The result indicates that the proposed framework can potentially perform well at detecting predefined unsafe actions in videos. Display omitted •The framework allows for the automatic detection of unsafe actions in site videos.•Computer vision techniques enable estimation of body joint positions on 2D images.•Using stereo cameras, a 2D human skeleton can be reconstructed in a 3D coordinate.•Dimension reduction can improve the accuracy and efficiency of motion analysis.•With motion templates, similar actions can be detected through pattern recognition.