Semantic segmentation, which aims to accurately identify each pixel, is a meaningful and challenging task. Recently, we witness a strong tendency to improve model efficiency in low-computing applications. However, most real-time methods ignore hierarchical features and context information to improve efficiency, leading to a decrease in the accuracy of semantic segmentation. To this end, we propose a novel system named hierarchical semantic-aware network (HSNet) to refine multilevel context information. HSNet mainly has the following two core modules: 1) hierarchical feature refinement module (HFRM) and 2) cross-scale pyramid fusion module (CPFM). By aggregating hierarchical feature maps, the proposed HFRM learns multilevel feature representation to recover spatial details. Afterward, the dual attention mechanism is developed to refine features from both channel and spatial levels, thereby alleviating the multilevel semantic gap. Meanwhile, the CPFM, which fuses local and global context information in a cross-scale manner, is proposed to enrich semantic information to improve accuracy. Furthermore, HSNet is carefully designed to improve the efficiency of the model by reusing shallow features and reducing channel capacity. Extensive experiments show that our method is effective and superior in segmentation accuracy and inference speed compared with state-of-the-art methods.