Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. ...To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the schema and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used datasets for images, videos, audios, and 3D data, as well as the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.
Noise suppression is an essential step in many seismic processing workflows. A portion of this noise, particularly in land datasets, presents itself as random noise. In recent years, neural networks ...have been successfully used to denoise seismic data in a supervised fashion. However, supervised learning always comes with the often unachievable requirement of having noisy-clean data pairs for training. Using blind-spot networks, we redefine the denoising task as a self-supervised procedure where the network uses the surrounding noisy samples to estimate the noise-free value of a central sample. Based on the assumption that noise is statistically independent between samples, the network struggles to predict the noise component of the sample due to its randomicity, whilst the signal component is accurately predicted due to its spatio-temporal coherency. Illustrated on synthetic examples, the blind-spot network is shown to be an efficient denoiser of seismic data contaminated by random noise with minimal damage to the signal; therefore, providing improvements in both the image domain and down-the-line tasks, such as post-stack inversion. To conclude our study, the suggested approach is applied to field data and the results are compared with two commonly used random denoising techniques: FX-deconvolution and sparsity-promoting inversion by Curvelet transform. By demonstrating that blind-spot networks are an efficient suppressor of random noise, we believe this is just the beginning of utilising self-supervised learning in seismic applications.
•Application of self-supervised deep learning for random noise suppression.•Blind-spot networks shown to effectively reduce white and coloured noise.•SNR improvements seen in the image, frequency and acoustic impedance domains.•Network successfully trained and applied to post-stack field data.•Blind-spot approach comparable to commonly used noise suppression procedures.
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted ...information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60 k hours to 94 k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.
In this paper, we propose a self-supervised video representation learning (video SSL) method by taking inspiration from cognitive science and neuroscience on human visual perception. Different from ...previous methods that focus on the inherent properties of videos, we argue that humans learn to perceive the world through the self-awareness of the semantic changes or consistency in the input stimuli in the absence of labels, accompanied by representation reorganization during the post-learning rest periods. To this end, we first exploit the presence of saccades as an indicator of semantic changes in a contrastive learning framework, mimicking self-awareness in human representation learning. The saccades are generated by alternating the fixations following the predicted scanpath. Second, we model the semantic consistency in eye fixation by minimizing the prediction error between the predicted and the true state of another time point. Finally, we incorporate prototypical contrastive learning to reorganize the learned representations to enhance the associations among perceptually similar ones. Compared to previous video SSL solutions, our method can capture finer-grained semantics from video instances and further associate similar ones together. Experiments show that the proposed bio-inspired video SSL method significantly improves the Top-1 video retrieval accuracy on UCF101 and achieves superior performance on downstream tasks such as action recognition under comparable settings.
Artificial intelligence (AI) is being increasingly integrated into scientific discovery to augment and accelerate research, helping scientists to generate hypotheses, design experiments, collect and ...interpret large datasets, and gain insights that might not have been possible using traditional scientific methods alone. Here we examine breakthroughs over the past decade that include self-supervised learning, which allows models to be trained on vast amounts of unlabelled data, and geometric deep learning, which leverages knowledge about the structure of scientific data to enhance model accuracy and efficiency. Generative AI methods can create designs, such as small-molecule drugs and proteins, by analysing diverse data modalities, including images and sequences. We discuss how these methods can help scientists throughout the scientific process and the central issues that remain despite such advances. Both developers and users of AI toolsneed a better understanding of when such approaches need improvement, and challenges posed by poor data quality and stewardship remain. These issues cut across scientific disciplines and require developing foundational algorithmic approaches that can contribute to scientific understanding or acquire it autonomously, making them critical areas of focus for AI innovation.
Large labeled datasets are crucial for video understanding progress. However, the labeling process is time-consuming, expensive, and tiresome. To overcome this impediment, various pretexts use the ...temporal coherence in videos to learn visual representations in a self-supervised manner. However, these pretexts (order verification and sequence sorting) struggle when encountering cyclic actions due to the label ambiguity problem. To overcome these limitations, we present a novel temporal pretext task to address self-supervised learning of visual representations from unlabeled videos. Repeated Scene Localization (RSL) is a multi-class classification pretext that involves changing the temporal order of the frames in a video by repeating a scene. Then, the network is trained to identify the modified video, localize the location of the repeated scene, and identify the unmodified original videos that do not have repeated scenes. We evaluated the proposed pretext on two benchmark datasets, UCF-101 and HMDB-51. The experimental results show that the proposed pretext achieves state-of-the-art results in action recognition and video retrieval tasks. In action recognition, our S3D model achieves 88.15% and 56.86% on UCF-101 and HMDB-51, respectively. It outperforms the current state-of-the-art by 1.05% and 3.26%. Our R(2+1)D-Adjacent model achieves 83.52% and 54.50% on UCF-101 and HMDB-51, respectively. It outperforms the single pretext tasks by 8.7% and 13.9%. In video retrieval, our R(2+1)D-Offset model outperforms the single pretext tasks by 4.68% and 1.1% Top 1 accuracies on UCF-101 and HMDB-51, respectively. The source code and the trained models are publicly available at https://github.com/Hussein-A-Hassan/RSL-Pretext.
•A novel temporal pretext is proposed for self-supervised representations learning.•It modifies the frames temporal order of an unlabeled video by repeating a scene.•A CNN is trained to identify the modified video and to localize the repeated scene.•The CNN identifies unmodified original videos that do not have repeated scenes.•The pretext is tested on action recognition and video retrieval using benchmark data.
Self-supervised learning (SSL) opens up huge opportunities for medical image analysis that is well known for its lack of annotations. However, aggregating massive (unlabeled) 3D medical images like ...computerized tomography (CT) remains challenging due to its high imaging cost and privacy restrictions. In our pilot study, we advocated bringing a wealth of 2D images like chest X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS. Especially, we designed a pyramid U- like medical Transformer (MiT) as the backbone to make UniMiSS possible to perform SSL with both 2D and 3D images. Consequently, the predecessor UniMiSS has two obvious merits compared to current 3D-specific SSL: (1) more effective - superior to learning strong representations, benefiting from more and diverse data; and (2) more versatile - suitable for various downstream tasks without the restriction on the dimensionality barrier. Unfortunately, UniMiSS did not dig deeply into the intrinsic anatomy correlation between 2D medical images and 3D volumes due to the lack of paired multi-modal/dimension patient data. In this extension paper, we propose the UniMiSS+, in which we introduce the digitally reconstructed radiographs (DRR) technology to simulate X-ray images from a CT volume to access paired CT and X-ray data. Benefiting from the paired group, we introduce an extra pair- wise constraint to boost the cross-modality correlation learning, which also can be adopted as a cross-dimension regularization to further improve the representations. We conduct expensive experiments on multiple 3D/2D medical image analysis tasks, including segmentation and classification. The results show that the proposed UniMiSS+ achieves promising performance on various downstream tasks, not only outperforming the ImageNet pre-training and other advanced SSL counterparts substantially but also improving the predecessor UniMiSS pre-training. Code is available at: https://github.com/YtongXie/UniMiSS-code .
Time-series representation learning is crucial for extracting meaningful representations from time-series data with temporal dynamics and sparse labels. Contrastive learning, a powerful technique for ...exploiting the inherent data patterns, has been applied to explore the diverse consistencies in time-series data, achieved through careful selection of contrastive pairs and design of appropriate loss. Encouraging such consistency is essential for acquiring comprehensive representations of time-series data. In this paper, we propose a new framework for time-series representation learning that combines the advantages of contextual, temporal, and transformation consistencies. This framework enables the network to learn general representations suitable for different tasks and domains. First, positive and negative pairs are generated to establish a multi-task learning setup. Then, contrastive losses are formulated to explore contextual, temporal, and transformation consistencies, which are jointly optimized to learn general time-series representations. In addition, we investigate an uncertainty weighting approach to enhance the effectiveness of multi-task learning. To evaluate the performance of our framework, we conduct experiments on three downstream tasks: time-series classification, forecasting, and anomaly detection. The experimental results demonstrate the superior performance of our framework compared to benchmark models across different tasks. Furthermore, our framework shows efficiency in cross-domain transfer learning scenarios.
•A new multi-task self-supervised time-series representation learning framework is proposed.•Our method efficiently intergrates contrastive learning approaches for contextual, temporal, and transformation consistency.•The proposed method can learn general representations for time-series classification, forecasting, and anomaly detection.
Purpose
To develop a model‐guided self‐supervised deep learning MRI reconstruction framework called reference‐free latent map extraction (RELAX) for rapid quantitative MR parameter mapping.
Methods
...Two physical models are incorporated for network training in RELAX, including the inherent MR imaging model and a quantitative model that is used to fit parameters in quantitative MRI. By enforcing these physical model constraints, RELAX eliminates the need for full sampled reference data sets that are required in standard supervised learning. Meanwhile, RELAX also enables direct reconstruction of corresponding MR parameter maps from undersampled k‐space. Generic sparsity constraints used in conventional iterative reconstruction, such as the total variation constraint, can be additionally included in the RELAX framework to improve reconstruction quality. The performance of RELAX was tested for accelerated T1 and T2 mapping in both simulated and actually acquired MRI data sets and was compared with supervised learning and conventional constrained reconstruction for suppressing noise and/or undersampling‐induced artifacts.
Results
In the simulated data sets, RELAX generated good T1/T2 maps in the presence of noise and/or undersampling artifacts, comparable to artifact/noise‐free ground truth. The inclusion of a spatial total variation constraint helps improve image quality. For the in vivo T1/T2 mapping data sets, RELAX achieved superior reconstruction quality compared with conventional iterative reconstruction, and similar reconstruction performance to supervised deep learning reconstruction.
Conclusion
This work has demonstrated the initial feasibility of rapid quantitative MR parameter mapping based on self‐supervised deep learning. The RELAX framework may also be further extended to other quantitative MRI applications by incorporating corresponding quantitative imaging models.
PDF
