Virtual Reality (VR) based training simulators are successfully employed in many industries (e.g., aviation) to help train operators and professionals in a safe environment. The construction industry has also started to use this technology in recent years for training operators of heavy equipment. However, the context presented in the available training simulators is unrealistic because in many instances the training takes place in static sites where there is no mobility in the site. To realistically introduce the context of construction sites into VR scenes sensory data from actual projects can be used. However, currently, there is no systematic insight into (1) the dimensions of context that need to be present in a training simulator, (2) the types of data required to represent various dimensions of the context, and (3) methods for converting context data into a coherent context-realistic training scene that enables bidirectional feedback between trainees and the VR scene. Therefore, this research aims to develop a novel framework to generate coherent context-realistic training simulators from data collected from actual construction projects to enhance construction training simulators. The proposed framework provides a step-wise guideline into (a) collection of appropriate data for context-realistic simulators, (b) development of agents and simulation physics from actual site data and their integration into a scene, (c) scene-trainee interactions in context-realistic scenes, and (d) context-based assessment of the trainees' performance from safety, productivity, and quality perspectives. A prototype is developed and a case study is conducted to demonstrate the feasibility of the proposed framework. A workshop with expert training instructors is conducted to evaluate the effectiveness of the proposed framework for improving simulator-based training. It is shown that compared to the existing simulators, the context-realistic training simulators can significantly improve various aspects of operator training, especially safety and teamwork. The research provided an insight into the future of construction training simulator by indicating the importance and relevance of (1) collecting appropriate data, and (2) developing robust data-to-agent and data-to-physics methods. •A framework for the context-realistic construction equipment training simulator is presented.•The framework integrates site layout data, mobility data, data-driven physics, and agent-based simulation.•Context-realistic simulators are shown to be promising for improving safety and teamwork of operator training.