Unmanned Aerial Vehicle (UAV) has seen a dramatic rise in popularity for remote-sensing image acquisition and analysis in recent years. It has brought promising results in low-altitude monitoring tasks that require detailed visual inspections. Semantic segmentation is one of the hot topics in UAV remote sensing image analysis, as its capability to mine contextual semantic information from UAV images is crucial for achieving a fine-grained understanding of scenes. However, in the remote sensing field, recent reviews have not focused on combining “UAV remote sensing” and “semantic segmentation” to summarize the advanced works and future trends. In this study, we focus primarily on describing various recent semantic segmentation methods applied in UAV remote sensing images and summarizing their advantages and limitations. According to the distinction in modeling contextual semantic information, we have categorized and outlined the methods based on graph-based contextual models and deep-learning-based models. Publicly available UAV-based image datasets are also gathered to encourage systematic research on advanced semantic segmentation methods. We provide quantitative results of representative methods on two high-resolution UAV-based image datasets for fair comparisons and discussions in terms of semantic segmentation accuracy and model inference efficiency. Besides, this paper concludes some remaining challenges and future directions in semantic segmentation for UAV remote sensing images and points out that methods based on deep learning will become the future research trend. •Summarize semantic segmentation methods and datasets for UAV remote sensing images.•Provide fair comparisons of representative methods in accuracy and efficiency.•Identify challenges and future directions for UAV image semantic segmentation.•Semantic segmentation methods based on deep learning will be the mainstream.