A review on deep learning in UAV remote sensing Osco, Lucas Prado; Marcato Junior, José; Marques Ramos, Ana Paula ...
International journal of applied earth observation and geoinformation,
October 2021, 2021-10-00, 2021-10-01, Letnik:
102
Journal Article
Recenzirano
Odprti dostop
•Combining deep learning and UAV-based data is an emerging trend in remote sensing.•Most articles published rely on CNN-based methods.•Future perspectives in UAV-based data processing still have much ...to cover.
Deep Neural Networks (DNNs) learn representation from data with an impressive capability, and brought important breakthroughs for processing images, time-series, natural language, audio, video, and many others. In the remote sensing field, surveys and literature revisions specifically involving DNNs algorithms’ applications have been conducted in an attempt to summarize the amount of information produced in its subfields. Recently, Unmanned Aerial Vehicle (UAV)-based applications have dominated aerial sensing research. However, a literature revision that combines both “deep learning” and “UAV remote sensing” thematics has not yet been conducted. The motivation for our work was to present a comprehensive review of the fundamentals of Deep Learning (DL) applied in UAV-based imagery. We focused mainly on describing the classification and regression techniques used in recent applications with UAV-acquired data. For that, a total of 232 papers published in international scientific journal databases was examined. We gathered the published materials and evaluated their characteristics regarding the application, sensor, and technique used. We discuss how DL presents promising results and has the potential for processing tasks associated with UAV-based image data. Lastly, we project future perspectives, commentating on prominent DL paths to be explored in the UAV remote sensing field. This revision consisting of an approach to introduce, commentate, and summarize the state-of-the-art in UAV-based image applications with DNNs algorithms in diverse subfields of remote sensing, grouping it in the environmental, urban, and agricultural contexts.
Deep neural networks are currently the focus of many remote sensing approaches related to forest management. Although they return satisfactory results in most tasks, some challenges related to ...hyperspectral data remain, like the curse of data dimensionality. In forested areas, another common problem is the highly-dense distribution of trees. In this paper, we propose a novel deep learning approach for hyperspectral imagery to identify single-tree species in highly-dense areas. We evaluated images with 25 spectral bands ranging from 506 to 820 nm taken over a semideciduous forest of the Brazilian Atlantic biome. We included in our network’s architecture a band combination selection phase. This phase learns from multiple combinations between bands which contributed the most for the tree identification task. This is followed by a feature map extraction and a multi-stage model refinement of the confidence map to produce accurate results of a highly-dense target. Our method returned an f-measure, precision and recall values of 0.959, 0.973, and 0.945, respectively. The results were superior when compared with a principal component analysis (PCA) approach. Compared to other learning methods, ours estimate a combination of hyperspectral bands that most contribute to the mentioned task within the network’s architecture. With this, the proposed method achieved state-of-the-art performance for detecting and geolocating individual tree-species in UAV-based hyperspectral images in a complex forest.
Image‐based gauging stations can allow for significant densification of monitoring networks of river water stages. However, thus far, most camera gauges do not provide the robustness of accurate ...measurements due to the varying appearance of water in the stream throughout the year. We introduce an approach that allows for automatic and reliable water stage measurement combining deep learning and photogrammetric techniques. First, a convolutional neural network (CNN), a class of deep learning, is applied to the segmentation (i.e., pixel classification) of water in images. The CNNs SegNet and fully convolutional network (FCN) are associated with a transfer learning strategy to segment water on images acquired by a Raspberry Pi camera. Errors of water segmentation with the two CNNs are lower than 3%. Second, the image information is transformed into metric water stage values by intersecting the extracted water contour, generated using the segmentation results, with a 3D model reconstructed with structure‐from‐motion (SfM) photogrammetry. The highest correlations between a reference gauge and the image‐based approaches reached 0.93, and average deviations were lower than 4 cm. Our approach allows for the densification of river monitoring networks based on camera gauges, providing accurate water stage measurements.
Key Points
Development of a camera gauge to measure the water stage from single images using convolutional neural network (CNN)
Combining automatic/robust water area detection with high‐resolution 3D data for stage retrieval in challenging environmental conditions
CNN based water stage measurements reveal accuracy potential at cm‐range and up to 93% agreement with traditional reference gauge
Visual inspection has been a common practice to determine the number of plants in orchards, which is a labor-intensive and time-consuming task. Deep learning algorithms have demonstrated great ...potential for counting plants on unmanned aerial vehicle (UAV)-borne sensor imagery. This paper presents a convolutional neural network (CNN) approach to address the challenge of estimating the number of citrus trees in highly dense orchards from UAV multispectral images. The method estimates a dense map with the confidence that a plant occurs in each pixel. A flight was conducted over an orchard of Valencia-orange trees planted in linear fashion, using a multispectral camera with four bands in green, red, red-edge and near-infrared. The approach was assessed considering the individual bands and their combinations. A total of 37,353 trees were adopted in point feature to evaluate the method. A variation of σ (0.5; 1.0 and 1.5) was used to generate different ground truth confidence maps. Different stages (T) were also used to refine the confidence map predicted. To evaluate the robustness of our method, we compared it with two state-of-the-art object detection CNN methods (Faster R-CNN and RetinaNet). The results show better performance with the combination of green, red and near-infrared bands, achieving a Mean Absolute Error (MAE), Mean Square Error (MSE), R2 and Normalized Root-Mean-Squared Error (NRMSE) of 2.28, 9.82, 0.96 and 0.05, respectively. This band combination, when adopting σ = 1 and a stage (T = 8), resulted in an R2, MAE, Precision, Recall and F1 of 0.97, 2.05, 0.95, 0.96 and 0.95, respectively. Our method outperforms significantly object detection methods for counting and geolocation. It was concluded that our CNN approach developed to estimate the number and geolocation of citrus trees in high-density orchards is satisfactory and is an effective strategy to replace the traditional visual inspection method to determine the number of plants in orchards trees.
Road extraction in remote sensing data: A survey Chen, Ziyi; Deng, Liai; Luo, Yuhua ...
International journal of applied earth observation and geoinformation,
August 2022, 2022-08-00, 2022-08-01, Letnik:
112
Journal Article
Recenzirano
Odprti dostop
•This review covers a wider perspective in terms of both 2D remote sensing images and 3D point clouds.•This review provides a detail survey on 2D and 3D remote sensing datasets used for road ...extraction.•This review presents a detail analysis of challenges and future trends of road extraction from remote sensing data.
Automated extraction of roads from remotely sensed data come forth various usages ranging from digital twins for smart cities, intelligent transportation, urban planning, autonomous driving, to emergency management. Many studies have focused on promoting the progress of methods for automated road extraction from aerial and satellite optical images, synthetic aperture radar (SAR) images, and LiDAR point clouds. In the past 10 years, no a more comprehensive survey on this topic could be found in literature. This paper attempts to provide a comprehensive survey on road extraction methods that use 2D earth observing images and 3D LiDAR point clouds. In this review, we first present a tree-structure that separate the literature into 2D and 3D. Then, further methodologies level classification is demonstrated both in 2D and 3D. In 2D and 3D, we introduce and analyze the literature published in the last ten years. Except for the methodologies, we also review the aspects of data commonly used. Finally, this paper explores the existing challenges and future trends.
•An approach to potentialize the Random Forest to predict maize-crop yield.•Random Forest algorithm performed better in tests with the ranking-based strategy.•Indices NDVI, NDRE, and GNDVI were that ...more contributed to maize yield prediction.
Random Forest (RF) is a machine learning technique that has been proved to be highly accurate in several agricultural applications. However, to yield prediction, how much this technique may be improved with the adoption of a ranking-based strategy is still an unknown issue. Here we propose a ranking-based approach to potentialize the RF method for maize yield prediction. This approach is based on the correlation parameter of individual vegetation indices (VIs). The VIs were individually ranked based on a merit metric that measures the improvement on the Pearson’s correlation coefficient by using RF against a baseline method. As a result, only the most relevant VIs were considered as input features to the RF model. We used 33 VIs extracted from multispectral UAV-based (unmanned aerial vehicle) imagery. The multispectral data were generated with two different sensors: Sequoia and MicaSense; during the 2017/2018 and 2018/2019 crop seasons, respectively. Amongst all the evaluated indices, NDVI, NDRE, and GNDVI were the top three in the ranking-based analysis, and their combination with RF increased the maize yield prediction. Our approach also outperformed other known machine learning methods, like support vector machine and artificial neural network. Additive regression, using the RF as the base weak learner, provided a higher accuracy with a correlation coefficient and MAE (Mean Absolute Error) of 0.78 and 853.11 kg ha−1, respectively. We conclude that the ranking-based strategy of VIs is appropriate to predict maize yield using machine learning methods and data derived from multispectral images. We demonstrated that our approach reduces the number of VIs needed to determine a high accuracy and relative low MAE, and the approach may contribute to decision-making actions, resulting in accurate management of maize fields.
Detection and classification of tree species from remote sensing data were performed using mainly multispectral and hyperspectral images and Light Detection And Ranging (LiDAR) data. Despite the ...comparatively lower cost and higher spatial resolution, few studies focused on images captured by Red-Green-Blue (RGB) sensors. Besides, the recent years have witnessed an impressive progress of deep learning methods for object detection. Motivated by this scenario, we proposed and evaluated the usage of Convolutional Neural Network (CNN)-based methods combined with Unmanned Aerial Vehicle (UAV) high spatial resolution RGB imagery for the detection of law protected tree species. Three state-of-the-art object detection methods were evaluated: Faster Region-based Convolutional Neural Network (Faster R-CNN), YOLOv3 and RetinaNet. A dataset was built to assess the selected methods, comprising 392 RBG images captured from August 2018 to February 2019, over a forested urban area in midwest Brazil. The target object is an important tree species threatened by extinction known as
Vogel (Fabaceae). The experimental analysis delivered average precision around 92% with an associated processing times below 30 miliseconds.
In recent years, many agriculture-related problems have been evaluated with the integration of artificial intelligence techniques and remote sensing systems. Specifically, in fruit detection ...problems, several recent works were developed using Deep Learning (DL) methods applied in images acquired in different acquisition levels. However, the increasing use of anti-hail plastic net cover in commercial orchards highlights the importance of terrestrial remote sensing systems. Apples are one of the most highly-challenging fruits to be detected in images, mainly because of the target occlusion problem occurrence. Additionally, the introduction of high-density apple tree orchards makes the identification of single fruits a real challenge. To support farmers to detect apple fruits efficiently, this paper presents an approach based on the Adaptive Training Sample Selection (ATSS) deep learning method applied to close-range and low-cost terrestrial RGB images. The correct identification supports apple production forecasting and gives local producers a better idea of forthcoming management practices. The main advantage of the ATSS method is that only the center point of the objects is labeled, which is much more practicable and realistic than bounding-box annotations in heavily dense fruit orchards. Additionally, we evaluated other object detection methods such as RetinaNet, Libra Regions with Convolutional Neural Network (R-CNN), Cascade R-CNN, Faster R-CNN, Feature Selective Anchor-Free (FSAF), and High-Resolution Network (HRNet). The study area is a highly-dense apple orchard consisting of Fuji Suprema apple fruits (Malus domestica Borkh) located in a smallholder farm in the state of Santa Catarina (southern Brazil). A total of 398 terrestrial images were taken nearly perpendicularly in front of the trees by a professional camera, assuring both a good vertical coverage of the apple trees in terms of heights and overlapping between picture frames. After, the high-resolution RGB images were divided into several patches for helping the detection of small and/or occluded apples. A total of 3119, 840, and 2010 patches were used for training, validation, and testing, respectively. Moreover, the proposed method’s generalization capability was assessed by applying simulated image corruptions to the test set images with different severity levels, including noise, blurs, weather, and digital processing. Experiments were also conducted by varying the bounding box size (80, 100, 120, 140, 160, and 180 pixels) in the image original for the proposed approach. Our results showed that the ATSS-based method slightly outperformed all other deep learning methods, between 2.4% and 0.3%. Also, we verified that the best result was obtained with a bounding box size of 160 × 160 pixels. The proposed method was robust regarding most of the corruption, except for snow, frost, and fog weather conditions. Finally, a benchmark of the reported dataset is also generated and publicly available.
Fish species recognition is an important task to preserve ecosystems, feed humans, and tourism. In particular, the Pantanal is a wetland region that harbors hundreds of species and is considered one ...of the most important ecosystems in the world. In this paper, we present a new method based on convolutional neural networks (CNNs) for Pantanal fish species recognition. A new CNN composed of three branches that classify the fish species, family and order is proposed with the aim of improving the recognition of species with similar characteristics. The branch that classifies the fish species uses information learned from the family and order, which has shown to improve the overall accuracy. Results on unrestricted image dataset showed that the proposed method provides superior results to traditional approaches. Our method obtained an accuracy of 0.873 versus 0.864 of traditional CNN in recognition of 68 fish species. In addition, our method provides fish family and order recognition, which obtained accuracies of 0.938 and 0.96, respectively. We hope that, with these promising results, an automatic tool can be developed to monitor species in an important region such as the Pantanal.
•In this paper, we propose a new method for Pantanal fish species recognition.•Taxonomy (fish order and family) was included in convolutional neural networks.•Experiments were performed on a image dataset with 68 fish species.•Results using the taxonomy proved superior to the traditional approach.
This study proposes and evaluates five deep fully convolutional networks (FCNs) for the semantic segmentation of a single tree species: SegNet, U-Net, FC-DenseNet, and two DeepLabv3+ variants. The ...performance of the FCN designs is evaluated experimentally in terms of classification accuracy and computational load. We also verify the benefits of fully connected conditional random fields (CRFs) as a post-processing step to improve the segmentation maps. The analysis is conducted on a set of images captured by an RGB camera aboard a UAV flying over an urban area. The dataset also contains a mask that indicates the occurrence of an endangered species called
Vogel, also known as cumbaru, taken as the species to be identified. The experimental analysis shows the effectiveness of each design and reports average overall accuracy ranging from 88.9% to 96.7%, an F1-score between 87.0% and 96.1%, and IoU from 77.1% to 92.5%. We also realize that CRF consistently improves the performance, but at a high computational cost.
PDF
