Visual data captured at construction sites is a rich source of information for the day-to-day operation of construction projects. The development of deep-learning-based methods has demonstrated their capabilities in analyzing complex visual data and inferring valuable insights. Recent applications of these methods in construction have also shown promising performance in making the construction management process smarter. To understand the current research trends and to highlight future research directions, this study reviews state-of-the-art deep-learning applications on visual data analytics in the context of construction project management. This in-depth review identifies six major fields and fifty-two subfields of construction management where deep-learning-based visual data analytics have been applied. It also proposes a generalized workflow for applying deep-learning-based visual data analytics methods for solving construction management problems. In addition, the study highlights three future research directions where deep-learning-based visual data analytics can be applied on relatively less explored 3D visual data. Display omitted •An in-depth review of 142 journal articles and conference papers related to deep learning-based visual data analytics in construction management applications.•Identification of 6 major fields and 52 subfields of construction management where deep learning-based visual data analytics were applied.•Analysis of the research field evolution since the inception of deep learning in 2012.•A generalized workflow for deploying deep learning-based visual data analytics for managing construction projects.•Knowledge gap and open challenges in each stage of the workflow.•Three future research directions where 3D visual data can be leveraged.