Hazards potentially affect the safety of people on construction sites include falls from heights (FFH), trench and scaffold collapse, electric shock and arc flash/arc blast, and failure to use proper personal protective equipment. Such hazards are significant contributors to accidents and fatalities. Computer vision has been used to automatically detect safety hazards to assist with the mitigation of accidents and fatalities. However, as safety regulations are subject to change and become more stringent prevailing computer vision approaches will become obsolete as they are unable to accommodate the adjustments that are made to practice. This paper integrates computer vision algorithms with ontology models to develop a knowledge graph that can automatically and accurately recognise hazards while adhering to safety regulations, even when they are subjected to change. Our developed knowledge graph consists of: (1) an ontological model for hazards: (2) knowledge extraction; and (3) knowledge inference for hazard identification. We focus on the detection of hazards associated with FFH as an example to illustrate our proposed approach. We also demonstrate that our approach can successfully detect FFH hazards in varying contexts from images. •A knowledge graph is developed to automatically identify hazards.•Computer vision algorithms and ontology are used to develop knowledge graph.•Examples are used to illustrate the feasibility of the proposed approach.