The huge amount of data currently produced by modern Earth Observation (EO) missions has allowed for the design of advanced machine learning techniques able to support complex Land Use/Land Cover ...(LULC) mapping tasks. The Copernicus programme developed by the European Space Agency provides, with missions such as Sentinel-1 (S1) and Sentinel-2 (S2), radar and optical (multi-spectral) imagery, respectively, at 10 m spatial resolution with revisit time around 5 days. Such high temporal resolution allows to collect Satellite Image Time Series (SITS) that support a plethora of Earth surface monitoring tasks. How to effectively combine the complementary information provided by such sensors remains an open problem in the remote sensing field. In this work, we propose a deep learning architecture to combine information coming from S1 and S2 time series, namely TWINNS (TWIn Neural Networks for Sentinel data), able to discover spatial and temporal dependencies in both types of SITS. The proposed architecture is devised to boost the land cover classification task by leveraging two levels of complementarity, i.e., the interplay between radar and optical SITS as well as the synergy between spatial and temporal dependencies. Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the Koumbia site in Burkina Faso and Reunion Island, a overseas department of France in the Indian Ocean), demonstrate the significance of our proposal.
Efficient methodologies for mapping croplands are an essential condition for the implementation of sustainable agricultural practices and for monitoring crops periodically. The increasing spatial and ...temporal resolution of globally available satellite images, such as those provided by Sentinel-2, creates new possibilities for generating accurate datasets on available crop types, in ready-to-use vector data format. Existing solutions dedicated to cropland mapping, based on high resolution remote sensing data, are mainly focused on pixel-based analysis of time series data. This paper evaluates how a time-weighted dynamic time warping (TWDTW) method that uses Sentinel-2 time series performs when applied to pixel-based and object-based classifications of various crop types in three different study areas (in Romania, Italy and the USA). The classification outputs were compared to those produced by Random Forest (RF) for both pixel- and object-based image analysis units. The sensitivity of these two methods to the training samples was also evaluated. Object-based TWDTW outperformed pixel-based TWDTW in all three study areas, with overall accuracies ranging between 78.05% and 96.19%; it also proved to be more efficient in terms of computational time. TWDTW achieved comparable classification results to RF in Romania and Italy, but RF achieved better results in the USA, where the classified crops present high intra-class spectral variability. Additionally, TWDTW proved to be less sensitive in relation to the training samples. This is an important asset in areas where inputs for training samples are limited.
Display omitted
•Time-weighted dynamic time warping (TWDTW) was evaluated for croplands mapping•Object-based TWDTW performed better than pixel-based TWDTW•TWDTW is robust in areas where inputs for training samples are limited•Automatic time-series Sentinel-2 data segmentation produced satisfactory results•Sentinel-2 proved to be a valuable satellite data source for croplands mapping
New remote sensing sensors will acquire High spectral, spatial and temporal Resolution Satellite Image Time Series (HR-SITS). These new data are of great interest to map land cover thanks to the ...combination of the three high resolutions that will allow a depiction of scene dynamics. However, their efficient exploitation involves new challenges, especially for adapting traditional classification schemes to data complexity. More specifically, it requires: (1) to determine which classifier algorithms can handle the amount and the variability of data; (2) to evaluate the stability of classifier parameters; (3) to select the best feature set used as input data in order to find the good trade-off between classification accuracy and computational time; and (4) to establish the classifier accuracy over large areas.
This work aims at studying these different issues, and more especially at demonstrating the ability of state-of-the-art classifiers, such as Random Forests (RF) or Support Vector Machines (SVM), to classify HR-SITS. For this purpose, several studies are carried out by using SPOT-4 and Landsat-8 HR-SITS in the south of France. Firstly, the choice of the classifier is discussed by comparing RF and SVM algorithms on HR-SITS. Both classifiers show their ability to tackle the classification problem with an Overall Accuracy (OA) of 83.3 % for RF and 77.1 % for SVM. But RF have some advantages such as a small training time, and an easy parameterization. Secondly, the stability of RF parameters is appraised. RF parameters appear to cause little influence on the classification accuracy, about 1% OA difference between the worst and the best parameter configuration. Thirdly, different input data – composed of spectral bands with or without spectral and/or temporal features – are proposed in order to enhance the characterization of land cover. The addition of features improves the classification accuracy, but the gain in OA is weak compared with the increase in the computational cost. Eventually, the classifier accuracy is assessed on a larger area where the landscape variabilities affect the classification performances.
•Similar classification performances are obtained between RF and SVM classifiers.•The setting of RF parameters has caused little influence on classification accuracy.•Classification performances are little increased by adding features with RF.•Classification accuracies are affected by the landscape diversity over large area.
Nowadays, modern Earth Observation systems continuously generate huge amounts of data. A notable example is represented by the Sentinel-2 mission, which provides images at high spatial resolution (up ...to 10 m) with high temporal revisit period (every 5 days), which can be organized in Satellite Image Time Series (SITS). While the use of SITS has been proved to be beneficial in the context of Land Use/Land Cover (LULC) map generation, unfortunately, most of machine learning approaches commonly leveraged in remote sensing field fail to take advantage of spatio-temporal dependencies present in such data.
Recently, new generation deep learning methods allowed to significantly advance research in this field. These approaches have generally focused on a single type of neural network, i.e., Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), which model different but complementary information: spatial autocorrelation (CNNs) and temporal dependencies (RNNs). In this work, we propose the first deep learning architecture for the analysis of SITS data, namely DuPLO (DUal view Point deep Learning architecture for time series classificatiOn), that combines Convolutional and Recurrent neural networks to exploit their complementarity. Our hypothesis is that, since CNNs and RNNs capture different aspects of the data, a combination of both models would produce a more diverse and complete representation of the information for the underlying land cover classification task. Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the Gard site in Mainland France and Reunion Island, a overseas department of France in the Indian Ocean), demonstrate the significance of our proposal.
•We present SITS-Former, which is the first pre-trained representation model for patch-based Sentinel-2 time series classification.•SITS-Former is pre-trained on massive unlabeled Sentinel-2 time ...series to learn spatio-spectral-temporal features via a missing-data imputation proxy task based on self-supervised learning.•SITS-Former can adapt the learned features to an interested classification task through fine-tuning.•SITS-Former outperforms state-of-the-art approaches and yields a significant improvement over the purely supervised method.
Sentinel-2 images provide a rich source of information for a variety of land cover, vegetation, and environmental monitoring applications due to their high spectral, spatial, and temporal resolutions. Recently, deep learning-based classification of Sentinel-2 time series becomes a popular solution to vegetation classification and land cover mapping, but it often demands a large number of manually annotated labels. Improving classification performance with limited labeled data is still a challenge in many real-world remote sensing applications. To address label scarcity, we present SITS-Former (SITS stands for Satellite Image Time Series and Former stands for Transformer), a pre-trained representation model for Sentinel-2 time series classification. SITS-Former adopts a Transformer encoder as the backbone and takes time series of image patches as input to learn spatio-spectral-temporal features. According to the principles of self-supervised learning, we pre-train SITS-Former on massive unlabeled Sentinel-2 time series via a missing-data imputation proxy task. Given an incomplete time series with some patches being masked randomly, the network is asked to regress the central pixels of these masked patches based on the residual ones. By doing so, the network can capture high-level spatial and temporal dependencies from the data to learn discriminative features. After pre-training, the network can adapt the learned features to a target classification task through fine-tuning. As far as we know, this is the first study that exploits self-supervised learning for patch-based representation learning and classification of SITS. We quantitatively evaluate the quality of the learned features by transferring them on two crop classification tasks, showing that SITS-Former outperforms state-of-the-art approaches and yields a significant improvement (2.64%∼3.30% in overall accuracy) over the purely supervised model. The proposed model provides an effective tool for SITS-related applications as it greatly reduces the burden of manual labeling. The source code will be released at https://github.com/linlei1214/SITS-Former upon publication.
In this article, we propose an approach based on Gaussian Processes (GP) for large scale land cover pixel-based classification with Sentinel-2 satellite image time-series (SITS). We used a sparse ...approximation of the posterior combined with variational inference to learn the GP's parameters. We applied stochastic gradient descent and GPU computing to optimize our GP models on massive data sets. The proposed GP model can be trained with hundreds of thousands of samples, compared to few thousands for traditional GP methods. Moreover, we included the spatial information by adding the geographic coordinates into the GP's covariance function to efficiently exploit the spatio-spectro-temporal structure of the SITS. We ran experiments with Sentinel-2 SITS of the full year 2018 over an area of 200 000 km 2 (about 2 billion pixels) in the south of France, which is representative of an operational setting. Adding the spatial information significantly improved the results in terms of classification accuracy. With spatial information, GP models have an overall accuracy of 79.8. They are more than three points above Random Forest (the method used for current operational systems) and more than one point above a multi-layer perceptron. Compared to a Transformer-based model (which provides state of the art results in the literature, but are not applied in operational systems), GP models are only one point below.
A detailed and accurate knowledge of land cover is crucial for many scientific and operational applications, and as such, it has been identified as an Essential Climate Variable. This accurate ...knowledge needs frequent updates. This paper presents a methodology for the fully automatic production of land cover maps at country scale using high resolution optical image time series which is based on supervised classification and uses existing databases as reference data for training and validation. The originality of the approach resides in the use of all available image data, a simple pre-processing step leading to a homogeneous set of acquisition dates over the whole area and the use of a supervised classifier which is robust to errors in the reference data. The produced maps have a kappa coefficient of 0.86 with 17 land cover classes. The processing is efficient, allowing a fast delivery of the maps after the acquisition of the image data, does not need expensive field surveys for model calibration and validation, nor human operators for decision making, and uses open and freely available imagery. The land cover maps are provided with a confidence map which gives information at the pixel level about the expected quality of the result.
The frequent acquisitions of fine spatial resolution imagery (10 m) offered by recent multispectral satellite missions, including Sentinel-2, can resolve single agricultural fields and thus provide ...crop-specific phenology metrics, a crucial information for crop monitoring. However, effective phenology retrieval may still be hampered by significant cloud cover. Synthetic aperture radar (SAR) observations are not restricted by weather conditions, and Sentinel-1 thus ensures more frequent observations of the land surface. However, these data have not been systematically exploited for phenology retrieval so far. In this study, we extracted crop-specific land surface phenology (LSP) from Sentinel-1 and Sentinel-2 of major European crops (common and durum wheat, barley, maize, oats, rape and turnip rape, sugar beet, sunflower, and dry pulses) using ground-truth information from the “Copernicus module” of the Land Use/Cover Area frame statistical Survey (LUCAS) of 2018. We consistently used a single model-fit approach to retrieve LSP metrics on temporal profiles of CR (Cross Ratio, the ratio of the backscattering coefficient VH/VV from Sentinel-1) and NDVI (Normalized Difference Vegetation Index from Sentinel-2). Our analysis revealed that LSP retrievals from Sentinel-1 are comparable to those of Sentinel-2, particularly for winter crops. The start of season (SOS) timings, as derived from Sentinel-1 and -2, are significantly correlated (average r of 0.78 for winter and 0.46 for summer crops). The correlation is lower for end of season retrievals (EOS, r of 0.62 and 0.34). Agreement between LSP derived from Sentinel-1 and -2 varies among crop types, ranging from r = 0.89 and mean absolute error MAE = 10 days (SOS of dry pulses) to r = 0.15 and MAE = 53 days (EOS of sugar beet). Observed deviations revealed that Sentinel-1 and -2 LSP retrievals can be complementary; for example for winter crops we found that SAR detected the start of the spring growth while multispectral data is sensitive to the vegetative growth before and during winter. To test if our results correspond reasonably to in-situ data, we compared average crop-specific LSP for Germany to average phenology from ground phenological observations of 2018 gathered from the German Meteorological Service (DWD). Our study demonstrated that both Sentinel-1 and -2 can provide relevant and at times complementary LSP information at field- and crop-level.
•Crop-specific phenology retrieved from Senetinel-1 and Sentinel-2 time series.•European Union wide survey LUCAS exploited to focus on major European crops.•Comparable phenology from cross polarization ratio VH/VV and from Sentinel-2 NDVI.•Retrieved phenology consistent with ground phenological observation from DWD.•Sentinel-1 and -2 provide relevant and at times complementary LSP information.
Display omitted
Precise crop mapping is crucial for guiding agricultural production, forecasting crop yield, and ensuring food security. Integrating optical and synthetic aperture radar (SAR) ...satellite image time series (SITS) for crop classification is an essential and challenging task in remote sensing. Previously published studies generally employ a dual-branch network to learn optical and SAR features independently, while ignoring the complementarity and correlation between the two modalities. In this article, we propose a novel method to learn optical and SAR features for crop classification through cross-modal contrastive learning. Specifically, we develop an updated dual-branch network with partial weight-sharing of the two branches to reduce model complexity. Furthermore, we enforce the network to map features of different modalities from the same class to nearby locations in a latent space, while samples from distinct classes are far apart, thereby learning discriminative and modality-invariant features. We conducted a comprehensive evaluation of the proposed method on a large-scale crop classification dataset. Experimental results show that our method consistently outperforms traditional supervised learning approaches, no matter the training samples are adequate or not. Our findings demonstrate that unifying the representations of optical and SAR image time series enables the network to learn more competitive features and suppress inference noise.
Nowadays, modern earth observation programs produce huge volumes of satellite images time series that can be useful to monitor geographical areas through time. How to efficiently analyze such a kind ...of information is still an open question in the remote sensing field. Recently, deep learning methods proved suitable to deal with remote sensing data mainly for scene classification(i.e., convolutional neural networks on single images) while only very few studies exist involving temporal deep learning approaches i.e., recurrent neural networks (RNNs) to deal with remote sensing time series. In this letter, we evaluate the ability of RNNs, in particular, the long short-term memory (LSTM) model, to perform land cover classification considering multitemporal spatial data derived from a time series of satellite images. We carried out experiments on two different data sets considering both pixel-based and object-based classifications. The obtained results show that RNNs are competitive compared with the state-of-the-art classifiers, and may outperform classical approaches in the presence of low represented and/or highly mixed classes. We also show that the alternative feature representation generated by LSTM can improve the performances of standard classifiers.
PDF
