Inspired by the ambitions envisioned in the Fourth Industrial Revolution for aquaculture, also known as Aquaculture 4.0, the aquaculture (marine animal farming) industry is seeking to adopt data-driven Artificial Intelligence (AI) to help significantly improve business operations. One of the major barriers is the manual annotation of animal behaviour data, which is a time-consuming task that demands high levels of concentration from biologists. To address this challenge, this paper proposes novel automatic animal behaviour monitoring tailored for industrial scenarios. Our approach introduces a real-time machine-learning-based instance segmentation system that is specialised for underwater environments, where large groups of shrimp are farmed. The implemented system achieves an accuracy rate of 89% at 30 frames per second (fps) and can accurately detect shrimp in high-density areas under poor lighting conditions and high turbidity waters, despite the challenges of occlusion and overlapping. A key innovation of our method is the implementation of a new density cluster algorithm for time series and video analysis. This approach provides a more efficient and accurate way of monitoring animal behaviour, significantly saving time and effort for biologists and advancing the capabilities of automated aquaculture systems. •Novel AI-system to enhance animal detection accuracy in high-density areas.•Enhanced DBScan algorithm for time series for density-based spatial clustering.•AI model with robustness in handling occlusion, turbidity, and overlapping.