Hyperspectral imagery collected from airborne or satellite sources inevitably suffers from spectral variability, making it difficult for spectral unmixing to accurately estimate abundance maps. The ...classical unmixing model, the linear mixing model (LMM), generally fails to handle this sticky issue effectively. To this end, we propose a novel spectral mixture model, called the augmented LMM, to address spectral variability by applying a data-driven learning strategy in inverse problems of hyperspectral unmixing. The proposed approach models the main spectral variability (i.e., scaling factors) generated by variations in illumination or typography separately by means of the endmember dictionary. It then models other spectral variabilities caused by environmental conditions (e.g., local temperature and humidity and atmospheric effects) and instrumental configurations (e.g., sensor noise), and material nonlinear mixing effects, by introducing a spectral variability dictionary. To effectively run the data-driven learning strategy, we also propose a reasonable prior knowledge for the spectral variability dictionary, whose atoms are assumed to be low-coherent with spectral signatures of endmembers, which leads to a well-known low-coherence dictionary learning problem. Thus, a dictionary learning technique is embedded in the framework of spectral unmixing so that the algorithm can learn the spectral variability dictionary and estimate the abundance maps simultaneously. Extensive experiments on synthetic and real datasets are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with the previous state-of-the-art methods.
This paper presents a hypserspectral image (HSI) super-resolution method, which fuses a low-resolution HSI (LR-HSI) with a high-resolution multispectral image (HR-MSI) to get high-resolution HSI ...(HR-HSI). The proposed method first extracts the nonlocal similar patches to form a nonlocal patch tensor (NPT). A novel tensor-tensor product (t - product)-based tensor sparse representation is proposed to model the extracted NPTs. Through the tensor sparse representation, both the spectral and spatial similarities between the nonlocal similar patches are well preserved. Then, the relationship between the HR-HSI and the LR-HSI is built using t - product, which allows us to design a unified objective function to incorporate the nonlocal similarity, tensor dictionary learning, and tensor sparse coding together. Finally, alternating direction method of multipliers is used to solve the optimization problem. Experimental results on three data sets and one real data set demonstrate that the proposed method substantially outperforms the existing state-ofthe-art HSI super-resolution methods.
Convolutional neural networks (CNNs) have been attracting increasing attention in hyperspectral (HS) image classification due to their ability to capture spatial-spectral feature representations. ...Nevertheless, their ability in modeling relations between the samples remains limited. Beyond the limitations of grid sampling, graph convolutional networks (GCNs) have been recently proposed and successfully applied in irregular (or nongrid) data representation and analysis. In this article, we thoroughly investigate CNNs and GCNs (qualitatively and quantitatively) in terms of HS image classification. Due to the construction of the adjacency matrix on all the data, traditional GCNs usually suffer from a huge computational cost, particularly in large-scale remote sensing (RS) problems. To this end, we develop a new minibatch GCN (called miniGCN hereinafter), which allows to train large-scale GCNs in a minibatch fashion. More significantly, our miniGCN is capable of inferring out-of-sample data without retraining networks and improving classification performance. Furthermore, as CNNs and GCNs can extract different types of HS features, an intuitive solution to break the performance bottleneck of a single model is to fuse them. Since miniGCNs can perform batchwise network training (enabling the combination of CNNs and GCNs), we explore three fusion strategies: additive fusion, elementwise multiplicative fusion, and concatenation fusion to measure the obtained performance gain. Extensive experiments, conducted on three HS data sets, demonstrate the advantages of miniGCNs over GCNs and the superiority of the tested fusion strategies with regard to the single CNN or GCN models. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_GCN for the sake of reproducibility.
With a large amount of open satellite multispectral (MS) imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global MS land cover classification. However, its limited ...spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to MS ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace), by jointly considering subspace learning and supervised classification. By locally aligning the manifold structure of the two modalities, CoSpace linearly learns a shared latent subspace from hyperspectral-MS (HS-MS) correspondences. The MS out-of-samples can be then projected into the subspace, which are expected to take advantages of rich spectral information of the corresponding hyperspectral data used for learning, and thus leads to a better classification. Extensive experiments on two simulated HS-MS data sets (University of Houston and Chikusei), where HS-MS data sets have tradeoffs between coverage and spectral resolution, are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.
The total variation (TV) regularization has been widely used in various applications related to hyperspectral (HS) signal and image processing due to its potential in modeling the underlying ...smoothness of HS data. However, most existing TV norms usually tend to generate spatial oversmoothing or artifacts. To this end, we propose a novel <inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula> hybrid TV (<inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula>HTV) regularization with the applications to HS mixed noise removal and compressed sensing (CS). More specifically, <inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula>HTV can be regarded as a globally and locally integrated TV regularizer, where the <inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula> gradient constraint is incorporate into the <inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula> spatial-spectral TV (<inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula>-SSTV). <inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula>-SSTV is capable of exploiting the local structure information across both spatial and spectral domains, while the <inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula> gradient can promote a globally spectral-spatial smoothness by directly controlling the number of nonzero gradients of HS images. This efficient combination considers more comprehensive prior knowledge of HS images, yielding sharper edge preservation and resolving the above drawbacks of existing pure TV norms. More significantly, <inline-formula> <tex-math notation="LaTeX">l_{0} </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">l_{1} </tex-math></inline-formula>HTV can be easily injected into HS-related processing models, and an effective algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the optimization problems. Extensive experiments conducted on several HS data sets substantiate the superiority and effectiveness of the proposed method in comparison with many state-of-the-art methods.
Spatiotemporal image fusion is considered as a promising way to provide Earth observations with both high spatial resolution and frequent coverage, and recently, learning-based solutions have been ...receiving broad attention. However, these algorithms treating spatiotemporal fusion as a single image super-resolution problem, generally suffers from the significant spatial information loss in coarse images, due to the large upscaling factors in real applications. To address this issue, in this paper, we exploit temporal information in fine image sequences and solve the spatiotemporal fusion problem with a two-stream convolutional neural network called StfNet . The novelty of this paper is twofold. First, considering the temporal dependence among image sequences, we incorporate the fine image acquired at the neighboring date to super-resolve the coarse image at the prediction date. In this way, our network predicts a fine image not only from the structural similarity between coarse and fine image pairs but also by exploiting abundant texture information in the available neighboring fine images. Second, instead of estimating each output fine image independently, we consider the temporal relations among time-series images and formulate a temporal constraint. This temporal constraint aiming to guarantee the uniqueness of the fusion result and encourages temporal consistent predictions in learning and thus leads to more realistic final results. We evaluate the performance of the StfNet using two actual data sets of Landsat-Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and both visual and quantitative evaluations demonstrate that our algorithm achieves state-of-the-art performance.
In recent years, enormous efforts have been made to design image-processing algorithms to enhance the spatial resolution of hyperspectral (HS) imagery. One of the most commonly addressed problems is ...the fusion of HS data with higher spatial resolution multispectral (MS) data. Various techniques have been proposed to solve this data-fusion problem based on different theories, including component substitution (CS), multiresolution analysis (MRA), spectral unmixing, and Bayesian probability. This article presents a comparative review of those HS-MS fusion techniques with extensive experiments. Ten state-of-the-art HS-MS fusion methods are compared by assessing their fusion performance both quantitatively and visually. Eight data sets featuring different geographical and sensor characteristics are used in the experiments to evaluate the generalizability and versatility of the fusion algorithms. To maximize the fairness and transparency of this comparison, publicly available source codes are used, and parameters are individually tuned for maximum performance.
With the rapid development of spaceborne imaging techniques, object detection in optical remote sensing imagery has drawn much attention in recent decades. While many advanced works have been ...developed with powerful learning algorithms, the incomplete feature representation still cannot meet the demand for effectively and efficiently handling image deformations, particularly objective scaling and rotation. To this end, we propose a novel object detection framework, called Optical Remote Sensing Imagery detector (ORSIm detector), integrating diverse channel features extraction, feature learning, fast image pyramid matching, and boosting strategy. An ORSIm detector adopts a novel spatial-frequency channel feature (SFCF) by jointly considering the rotation-invariant channel features constructed in the frequency domain and the original spatial channel features (e.g., color channel and gradient magnitude). Subsequently, we refine SFCF using learning-based strategy in order to obtain the high-level or semantically meaningful features. In the test phase, we achieve a fast and coarsely scaled channel computation by mathematically estimating a scaling factor in the image domain. Extensive experimental results conducted on the two different airborne data sets are performed to demonstrate the superiority and effectiveness in comparison with the previous state-of-the-art methods.
Hyperspectral remote sensing images (HSIs) usually have high spectral resolution and low spatial resolution. Conversely, multispectral images (MSIs) usually have low spectral and high spatial ...resolutions. The problem of inferring images that combine the high spectral and high spatial resolutions of HSIs and MSIs, respectively, is a data fusion problem that has been the focus of recent active research due to the increasing availability of HSIs and MSIs retrieved from the same geographical area. We formulate this problem as the minimization of a convex objective function containing two quadratic data-fitting terms and an edge-preserving regularizer. The data-fitting terms account for blur, different resolutions, and additive noise. The regularizer, a form of vector total variation, promotes piecewise-smooth solutions with discontinuities aligned across the hyperspectral bands. The downsampling operator accounting for the different spatial resolutions, the nonquadratic and nonsmooth nature of the regularizer, and the very large size of the HSI to be estimated lead to a hard optimization problem. We deal with these difficulties by exploiting the fact that HSIs generally "live" in a low-dimensional subspace and by tailoring the split augmented Lagrangian shrinkage algorithm (SALSA), which is an instance of the alternating direction method of multipliers (ADMM), to this optimization problem, by means of a convenient variable splitting. The spatial blur and the spectral linear operators linked, respectively, with the HSI and MSI acquisition processes are also estimated, and we obtain an effective algorithm that outperforms the state of the art, as illustrated in a series of experiments with simulated and real-life data.
With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity are readily available ...nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyze and interpret strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the field of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More specifically, some essential knowledge about this topic is first given. Subsequently, a literature survey is conducted to analyze the trends of this field. Some prevalent sub-fields in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.
•A systematic review of deep learning-based multimodal remote sensing data fusion.•Statistical analysis of relevant literature is conducted.•Seven prevalent sub-fields in multimodal remote sensing data fusion are detailed.•Some available resources, including tutorials, datasets, and codes, are provided.•Deep learning yields great achievements in multimodal remote sensing data fusion.
PDF
