Display omitted •An effective metric is proposed to quantify the maceral composition of coal.•The task of maceral group identification is defined as a semantic segmentation task.•The proposed end-to-end way bypasses the steps limiting the traditional strategies. Correct identification macerals is important for analyzing petrographic characteristics of coal. Traditional methods based on manual measurement are time-consuming and physically demanding. Thus, automation of this process is highly desirable. However, currently available machine learning-based methods mostly depend on handcrafted features, and do not work in analyzing complex samples. There is still much room to enhance the robustness and generalization ability for these automatic methods. To address these issues, we interpret the task of maceral group identification as a semantic segmentation task. In this study, we design an improved U-net model with enhanced attention gates, and evaluate the performance with various encoder backbones. Experimental results on 89 photomicrographs indicate that the proposed method achieves the state-of-the-art segmentation accuracy of 91.56%. The model achieves mean absolute errors of 5.95%, 4.43% and 2.19% for the predicted proportion of vitrinite, inertinite and liptinite, respectively. It shows that the improved U-net model has significant potential for automating accurate identification of maceral groups.