•Cross-modal data retrieval.•Novel framework for designing a unified representation for different modalities of data.•Improved performance using the unified representation for both cross and ...uni-modal retrieval.•Extensive experiments performed on the benchmark DSRSID and UC-Merced with audio samples datasets.
We address the problem of cross-modal information retrieval in the domain of remote sensing. In particular, we are interested in two application scenarios: i) cross-modal retrieval between panchromatic (PAN) and multispectral imagery, and ii) multi-label image retrieval between very high resolution (VHR) images and speech-based label annotations. These multi-modal retrieval scenarios are more challenging than the traditional uni-modal retrieval approaches given the inherent differences in distributions between the modalities. However, with the increasing availability of multi-source remote sensing data and the scarcity of enough semantic annotations, the task of multi-modal retrieval has recently become extremely important. In this regard, we propose a novel deep neural network-based architecture that is considered to learn a discriminative shared feature space for all the input modalities, suitable for semantically coherent information retrieval. Extensive experiments are carried out on the benchmark large-scale PAN - multispectral DSRSID dataset and the multi-label UC-Merced dataset. Together with the Merced dataset, we generate a corpus of speech signals corresponding to the labels. Superior performance with respect to the current state-of-the-art is observed in all the cases.
Hyperspectral images (HSIs) showing objects belonging to several distinct target classes are characterized by dozens of spectral bands being available. However, some of these spectral bands are ...redundant and/or noisy, and hence, selecting highly informative and trustworthy bands for each class is a vital step for classification and for saving internal storage space; then the selected bands are termed a highly informative spectral band subset. We use a mutual information (MI)-based method to select the spectral band subset of a given class and two additional binary quantum classifiers, namely a quantum boost (Qboost) and a quantum boost plus (Qboost-Plus) classifier, to classify a two-label dataset characterized by the selected band subset. We pose both our MI-based band subset selection problem and the binary quantum classifiers as a quadratic unconstrained binary optimization (QUBO) problem. Such a quadratic problem is solvable with the help of conventional optimization techniques. However, the QUBO problem is an NP-hard global optimization problem, and hence, it is worthwhile for applying a quantum annealer. Thus, we adapted our MI-based optimization problem for selecting highly informative bands for each class of a given HSI to be run on a D-Wave quantum annealer. After the selection of these highly informative bands for each class, we employ our binary quantum classifiers to a two-label dataset on the D-Wave quantum annealer. In addition, we provide a novel multilabel classifier exploiting an error-encoding output code when using our binary quantum classifiers. As a real-world dataset in Earth observation, we used the well-known AVIRIS HSI of Indian Pine, north-western Indiana, USA. We can demonstrate that the MI-based band subset selection problem can be run on a D-Wave quantum annealer that selects the highly informative spectral band subset for each target class in the Indian Pine HSI. We can also prove that our binary quantum classifiers and our novel multilabel classifier generate a correct two- and multilabel dataset characterized by their selected bands and with high accuracy such as having been produced by conventional classifiers-and even better in some instances.
Optical remote sensing instruments accumulate abundant data from across all of the earth's land surfaces, making it possible both to understand the effects of climate change and to monitor, ...investigate, and manage ground-level events in detail. Processing data using resources located near on-board satellite sensors can bring major benefits in terms of minimizing analysis time and quickly initiating active actions in critical situations. In satellite missions, long-term production on-board algorithms may encounter unexplored samples, i.e., abnormal ground-level events, and need to be able to discriminate and take the correct action. In this matter, the authors present a field programmable gate array (FPGA)-based solution for natural anomaly detection in multispectral imagery using deep convolutional neural networks. The effects of weather-induced hazards and natural disasters, considered anomalies in this sense, are discovered by modeling an anomaly detector on a hybrid system that is hardware efficient. The proposed approach is assembled on a Xilinx Zynq UltraScale+ XCZU9EG multiprocessor system-on-chip (MPSoC) device, where a deep convolutional model is scaled into the FPGA logic, followed by a downstream statistical meta-recognition predictor. The proposed anomaly detection accelerator has produced notable results in identifying a contemporary natural hazard, i.e., burned areas, in scenes acquired by Sentinel-2 over Europe, i.e., Spain and France. The implemented algorithm achieved on the FPGA accelerator an equivalent speedup of 4.46× and 4.5× lower power consumption than the equivalent implementation on the Tesla K80 GPU.
Recent advances in remote sensing technology have provided (very) high spatial resolution Earth Observation data with abundant latent semantic information. Conventional data processing algorithms are ...not capable of extracting the latent semantic information form these data and harness their full potential. As a result, semantic information discovery methods, based on data mining techniques, such as latent Dirichlet allocation and bag of visual words models, can discover the latent information. Despite their crucial rule, there are only a few studies in the field of semantic data mining for remote sensing applications. This article is focused on this shortage. Three different scenarios are used to evaluate the semantic information discovery in various remote sensing applications, including both optical and synthetic aperture radar (SAR) data with different spatial resolutions. In the first scenario, semantic discovery method correlated the semantic perception of the user and machine to correct and enhance the user defined Ground Truth map in very high-resolution RGB data. The potential of the semantic discovery is evaluated for wildfire affected area detection in Sentinel-2 data in the second scenario. Finally, in the third scenario, the semantic discovery method is utilized to detect the misclassifications as well as the patches with ambiguous or multiple semantic labels in a Sentinel-1 SAR patch-based benchmark dataset to enhance the robustness and accuracy of the annotation in the dataset. Our results in these three scenarios demonstrated the capability of the data-mining-based semantic information discovery methods for various remote sensing.
This paper presents a novel framework for multilabel classification of multispectral remote sensing images using error-correcting output codes. Starting with a set of primary class labels, the ...proposed framework consists in transforming the multiclass problem into multiple binary learning subtasks. The distributed output representations of these binary learners are then transformed into primary class labels. In order to train robust binary classifiers on a reduced annotated dataset, the learning process is iterative and involves determining most ambiguous examples, which are included in the training set at each iteration. As part of the semantic image recognition process, two categories of high-level image representations are proposed for the feature extraction part. First, deep convolutional neural networks are used to form high-level representations of the images. Second, we test our classification framework with a bag-of-visual words model based on the scale invariant feature transform, used in combination with color descriptors. In the first case, we propose the usage of pretrained state-of-the-art deep learning models that cancel the need to estimate model parameters of complex architectures, whereas, in the second case, a dictionary of visual words must be determined from the training set. Experiments are conducted on GeoEye-1 and Sentinel-2 images and the results show the effectiveness of the proposed approach toward a multilabel classification, when compared to other methods.
In this letter, we are interested in the annotation of large satellite images, using semantic concepts defined by the user. This annotation task combines a step of supervised classification of ...patches of the large image and the integration of the spatial information between these patches. Given a training set of images for each concept, learning is based on the latent Dirichlet allocation (LDA) model. This hierarchical model represents each item of a collection as a random mixture of latent topics, where each topic is characterized by a distribution over words. The LDA-based image representation is obtained using simple features extracted from image words. We then exploit the capability of the LDA model to assign probabilities to unseen images, in order to classify the patches of the large image into the semantic concepts, using the maximum-likelihood method. We conduct experiments on panchromatic QuickBird images with 60-cm resolution. Taking into account the spatial information between the patches shows to improve the annotation performance.
This article studies remote sensing image retrieval using kernel-based support vector data description (SVDD). We exploit deep SVDD, which is a well-known method for one-class classification to ...recover the most relevant samples from the archive. To this end, a deep neural network (DNN) is jointly trained to map the data into a hypersphere of minimum volume in the latent space. It is expected that similar samples to the query are compressed inside of the hypersphere. The closest embedding to the center of the hypersphere is related to the most relevant sample to query. We enhance deep SVDD by injecting the statistical information of data to the DNN by means of additional terms in the cost function. The first enhancement method takes advantage of covariance regularization of batches of the training set to penalize unnecessary redundancy and minimize the correlation between the different dimensions of the embedding. The second method involves unlocking the hypersphere's predefined center while preventing network divergence during training. Therefore, two parameters are designed to control the importance of the drifting of the center and the importance of a fixed predefined center (convergence), respectively. This has been implemented by considering the average of batches of embedding in each iteration as the updated center. This pushes irrelevant samples away from query samples, making data clustering easier for the DNN. The performance of the proposed methods is evaluated on benchmark datasets.
The large volume of detailed land cover features, provided by high resolution Earth observation (EO) images, has attracted considerable interest in the discovery of these features by learning ...systems. In this letter, we perform latent Dirichlet allocation on the bag of words (BoW) representation of collections of EO image patches to discover their semantic-level features, the so-called topics. To assess the discovered topics, the images are represented based on the occurrence of different topics, called bag of topics (BoT). The value added by BoT to the BoW model of image patches is then measured based on existing human annotations of the data. In our experiments, we compare the classification accuracy results of BoT and BoW representations of two different remote sensing image data sets, a multispectral optical data set and a synthetic-aperture-radar data set. Experimental results demonstrate that BoT can provide a compact and semantically meaningful representation of data; it either causes no significant reduction in the classification accuracy or increases the accuracy by a sufficient number of topics.
As deep learning attracts earth observation (EO) community's interest, the challenge to derive explainable, actionable information creates a bottleneck in EO models' development. Computer vision ...proved that effective results of the DL algorithms imply a considerable amount of training datasets. This is not the case in EO, where images are characterized by a broad variety of sensor data, ranging from multispectral to Synthetic Aperture Radar (SAR) with variable number of spectral bands, polarization, or spatial resolution. In this article, we present effective methodologies for fast training with reduced datasets of simple deep neural networks, while preserving the similar performance of state-of-the-art methods. The hybrid solutions we provide imply the reduction of input data dimension in a convolutional neural network. We replaced dataset's patches with histograms of pixel intensity, Bag-of-Words, or downsampled images. Following the proposed approaches, the training time and the dataset size are significantly reduced, while the performance of classification is preserved. These optimized implementations enable the deployment of lightweight deep learning models for real-time processing tasks able to exhibit accurate results, for instance, a disaster management scenario. We demonstrated the computational efficiency of these approaches on various, complex data, both multispectral and SAR, with different resolutions.
The Sentinel-1 mission provides a freely accessible opportunity for urban image interpretation based on synthetic aperture radar (SAR) data with a specific resolution, which is of paramount ...importance for Earth observation. In parallel, with the rapid development of advanced technologies, especially deep learning, we urgently need a large-scale SAR dataset supporting urban image interpretation. This article presents OpenSARUrban: a Sentinel-1 dataset dedicated to the content-related interpretation of urban SAR images, including a well-defined hierarchical annotation scheme, data collection, well-established procedures for dataset compilation and organization as well as properties, visualizations, and applications of this dataset. Particularly, our OpenSARUrban collection provides 33 358 image patches of urban SAR scenes, covering 21 major cities of China, including 10 different target area categories, 4 kinds of data formats, 2 kinds of polarization modes, and owning 5 essential properties: large-scale coverage, diversity, specificity, reliability, and sustainability. These properties guarantee the achievement of several goals for OpenSARUrban. The first one is to support urban target characterization. The second one is to help develop well-applicable and advanced algorithms for Sentinel-1 urban target classification. The third one is to explore content-based image retrieval for these kinds of data. In addition, dataset visualization is implemented from the perspective of manifolds to give an intuitive understanding. Besides a detailed description and visualization of the dataset, we present results of some benchmarking algorithms, demonstrating that this dataset is practical and challenging. Notably, developing algorithms to enhance the classification performance on the whole dataset and considering the data imbalance are especially demanding.
PDF
