The emergence of Internet of Things (IoT) technologies in the field of health monitoring has introduced the paradigm of Industrial Internet of Things (IIoT) to the industry. IIoT systems provide ...enterprises with a substantial volume of monitoring data for industrial equipment health monitoring, facilitating the development of artificial intelligence fault diagnosis models. However, a singular industrial entity often encounters limitations in collecting sufficient training data in practical scenarios. Moreover, the sharing of confidential information among entities is strictly prohibited due to concerns regarding intellectual property and data security. This study proposes a fault diagnosis system that addresses this issue by incorporating a capsule-based fault feature expression into the federated learning (FL) framework. The system comprises clients distributed across multiple factories and a central server hosted in the cloud. The client models are trained on local private datasets, and then knowledge fusion is achieved by uploading intrinsic templates and pose matrices to the central server. The proposed method offers the advantage of reducing transmission burden and enhancing data security in comparison to existing FL approaches. Besides, a capsule knowledge alignment algorithm is proposed to update the capsule-based fault feature expression ona central server. To simulate real fault diagnosis application scenarios, two similar fault simulation platforms are built to acquire isolated fault diagnosis datasets. The effectiveness of the proposed method is verified using these datasets.
•A federated fault diagnosis system is proposed for precise and secure fault diagnosis.•The proposed framework mitigates communication overhead and enhance data security.•A capsule knowledge alignment algorithm is proposed for updating central model.•A two-stage clustering method is proposed for integrating capsule knowledge.•The proposed method is validated on real-world distributed private local data.
•The difficulties that exist in conventional image fusion research are analyzed.•The advantages of deep learning (DL) techniques for image fusion are discussed.•A detailed review of existing DL-based ...image fusion methods is presented.•Several generic frameworks for DL-based image fusion are summarized and presented.•Some prospects for the future study of DL-based image fusion are put forward.
By integrating the information contained in multiple images of the same scene into one composite image, pixel-level image fusion is recognized as having high significance in a variety of fields including medical imaging, digital photography, remote sensing, video surveillance, etc. In recent years, deep learning (DL) has achieved great success in a number of computer vision and image processing problems. The application of DL techniques in the field of pixel-level image fusion has also emerged as an active topic in the last three years. This survey paper presents a systematic review of the DL-based pixel-level image fusion literature. Specifically, we first summarize the main difficulties that exist in conventional image fusion research and discuss the advantages that DL can offer to address each of these problems. Then, the recent achievements in DL-based image fusion are reviewed in detail. More than a dozen recently proposed image fusion methods based on DL techniques including convolutional neural networks (CNNs), convolutional sparse representation (CSR) and stacked autoencoders (SAEs) are introduced. At last, by summarizing the existing DL-based image fusion methods into several generic frameworks and presenting a potential DL-based framework for developing objective evaluation metrics, we put forward some prospects for the future study on this topic. The key issues and challenges that exist in each framework are discussed.
Background
Deep learning (DL) has been widely used for diagnosis and prognosis prediction of numerous frequently occurring diseases. Generally, DL models require large datasets to perform accurate ...and reliable prognosis prediction and avoid overlearning. However, prognosis prediction of rare diseases is still limited owing to the small number of cases, resulting in small datasets.
Purpose
This paper proposes a multimodal DL method to predict the prognosis of patients with malignant pleural mesothelioma (MPM) with a small number of 3D positron emission tomography–computed tomography (PET/CT) images and clinical data.
Methods
A 3D convolutional conditional variational autoencoder (3D‐CCVAE), which adds a 3D‐convolutional layer and conditional VAE to process 3D images, was used for dimensionality reduction of PET images. We developed a two‐step model that performs dimensionality reduction using the 3D‐CCVAE, which is resistant to overlearning. In the first step, clinical data were input to condition the model and perform dimensionality reduction of PET images, resulting in more efficient dimension reduction. In the second step, a subset of the dimensionally reduced features and clinical data were combined to predict 1‐year survival of patients using the random forest classifier. To demonstrate the usefulness of the 3D‐CCVAE, we created a model without the conditional mechanism (3D‐CVAE), one without the variational mechanism (3D‐CCAE), and one without an autoencoder (without AE), and compared their prediction results. We used PET images and clinical data of 520 patients with histologically proven MPM. The data were randomly split in a 2:1 ratio (train : test) and three‐fold cross‐validation was performed. The models were trained on the training set and evaluated based on the test set results. The area under the receiver operating characteristic curve (AUC) for all models was calculated using their 1‐year survival predictions, and the results were compared.
Results
We obtained AUC values of 0.76 (95% confidence interval CI, 0.72–0.80) for the 3D‐CCVAE model, 0.72 (95% CI, 0.68–0.77) for the 3D‐CVAE model, 0.70 (95% CI, 0.66–0.75) for the 3D‐CCAE model, and 0.69 (95% CI 0.65–0.74) for the without AE model. The 3D‐CCVAE model performed better than the other models (3D‐CVAE, p = 0.039; 3D‐CCAE, p = 0.0032; and without AE, p = 0.0011).
Conclusions
This study demonstrates the usefulness of the 3D‐CCVAE in multimodal DL models learned using a small number of datasets. Additionally, it shows that dimensionality reduction via AE can be used to learn a DL model without increasing the overlearning risk. Moreover, the VAE mechanism can overcome the uncertainty of the model parameters that commonly occurs for small datasets, thereby eliminating the risk of overlearning. Additionally, more efficient dimensionality reduction of PET images can be performed by providing clinical data as conditions and ignoring clinical data‐related features.
The interactions among the gauged data in most exiting real-life cases are correlative inevitably given the complicated behavior of process systems, that is the observed input data should better be ...interpreted as generating from joint interaction of static and dynamic feature sources. Therefore, the traditional single static-based or dynamic-based methods will inevitably lose some of the important information features, which will result in unsatisfactory monitoring results. Motivated by this, we expect to divide the data into static and dynamic features to perform separate modeling. Hence, the kernel slow feature analysis method is proposed to achieve this purpose. The output from the KSFA model can be divided into static and dynamic sources according to the dynamic characteristics. Then, in order to extract the unsupervised dynamic features, a novel deep RNN-LSTM autoencoder model is developed, which includes three components: data conversion, deep model construction, and feature removal. Moreover, we use a stacked autoencoder for static sources to learn static features and construct high-order models for fault detection. Based on the well-built detection system formed by the joint deep models, dual-scale decision-making is integrated by the Bayesian inference method. Finally, the superiority of the proposal can be shown in two processes.
Anomaly detection is indispensable for ensuring the reliable operation of grid-connected photovoltaic (PV) systems. This study introduces a semi-supervised deep learning approach for fault detection ...in such systems. The method leverages a variational autoencoder (VAE) to extract features and identify anomalies. By training the VAE on normal operation data, a compact latent space representation is created. Abnormal observations, indicating faults, exhibit distinct feature vectors in this latent space. Multiple anomaly detection algorithms, including Isolation Forest, Epileptic Envelope, Local Outlier Factor, and One-Class SVM, are employed to discern normal and abnormal observations. This semi-supervised approach only requires fault-free data for training, without labeled faults, making it attractive in practice. A publicly available dataset, the Grid-connected PV System Faults (GPVS-Faults) dataset, which includes data from a PV plant operating in both maximum power point tracking (MPPT) and intermediate power point tracking (IPPT) switching modes, is used for evaluation. The proposed approach is assessed across various fault scenarios, such as partial shading, inverter faults, and MPPT/IPPT controller faults in boost converters. The outcomes underscore the effectiveness of VAE-based techniques in accurately identifying these faults, with accuracy rates reaching up to 92.90% for MPPT mode and 92.99% for IPPT mode, thus contributing to the robustness of fault detection in grid-connected PV systems.
•Anomaly detection is crucial for reliable grid-connected PV systems.•Developed semi-supervised anomaly detection for reliable operation of PV systems.•Introduced Variational Autoencoder (VAE) for feature extraction and anomaly detection.•Effectiveness of VAE-based methods in detecting faults under MPPT and IPPT modes, achieving high AUC and F1-scores.•A publicly available dataset, GPVS-Faults, validate the VAE-based anomaly detectors.
In recent years, researchers have extensively explored the application of drive-by inspection technology for bridge damage assessment. This approach involves using the response of a sensing vehicle ...to identify damage. However, many existing methods rely on data collected from both healthy and damaged bridge conditions, which may not always be available. Therefore, this study introduces a fully unsupervised computer vision-based methodology for bridge structural health monitoring (BSHM) using drive-by inspection. It analyzes the time–frequency domain of a two-axle vehicle’s response by deriving a novel formulation for the contact point response from vehicle axles. The axles signals are then processed through subtraction, filtering, and decomposition using empirical Fourier decomposition with an improved segmentation approach based on the Savitzky–Golay filter (SGEFD). Relevant Intrinsic Mode Functions (IMF) are extracted as features representing damage, and the Wavelet Synchro-squeezed transform (WSST) is obtained from these features and used as input for the damage assessment algorithm. The performance of two state-of-the-art unsupervised generative machine learning methods, namely convolutional variational autoencoders (CVAE) and convolutional adversarial autoencoders (CAAE), is compared for the damage assessment task. These methods are trained solely with the residual WSST obtained from the vehicle responses when traversing a bridge in its reference state. A damage index (DI) is defined based on the measured error between the original and reconstructed images, and a damage threshold is calculated from the DI distribution of samples from the benchmark bridge state. During testing, the error between the original and reconstructed WSST is compared to the damage threshold, enabling the classification of new samples as healthy or damaged. The methodology is evaluated using both numerical and experimental vehicle–bridge interaction (VBI) models, considering various damage locations, severities, and the influence of road roughness.
Rolling bearings are a critical component of mechanical transmission equipment. Predicting their degradation trend is crucial for ensuring safe and stable equipment operation. Most existing bearing ...degradation prediction methods based on state space models (SSMs) use either linear functions or limited nonlinear functions (e.g., exponential/power laws) to construct the state and measurement equations. As such, these models fail to adapt to the complex and varied nonlinear degradation processes that occur in real-world environments. To address this limitation, we developed a deep latent variable-driven state space degradation model and employed it for bearing degradation prediction. Owing to the powerful nonlinear modeling ability of deep learning models, the proposed method extends the applicability of state space equations. In addition, it inherits the advantages of SSMs and can model uncertainties in a structured manner. Furthermore, the model was integrated with differential pre-transformation to improve its long-term prediction performance. Finally, to validate the effectiveness of the proposed model in predicting bearing degradation, experiments were conducted using a bearing dataset from the PRONOSTIA platform and real wind turbine bearing data. Results showed that the proposed method yielded higher bearing degradation prediction accuracy than existing methods, thus demonstrating the superior performance of the proposed model in predicting bearing degradation.
High labor costs and the requirement for significant domain expertise often result in a lack of anomaly labels in most time series. Consequently, employing unsupervised methods becomes critical for ...practical industrial applications. However, prevailing reconstruction-based anomaly detection algorithms encounter challenges in capturing intricate underlying correlations and temporal dependencies in time series. This study introduces an unsupervised anomaly detection model called Variational AutoeEncoder with Adversarial Training for Multivariate Time Series Anomaly Detection (VAEAT). Its fundamental concept involves adopting a two-phase training strategy to improve anomaly detection precision through adversarial reconstruction of raw data. In the first phase, the model reconstructs raw data to extract its basic features by training two enhanced variational autoencoders (VAEs) that incorporate both the long short-term memory (LSTM) network and the attention mechanism in their common encoder. In the second phase, the model refines reconstructed data to optimize the reconstruction quality. In this manner, this two-phase VAE model effectively captures intricate underlying correlation and temporal dependencies. A large number of experiments are conducted to evaluate the performance on five publicly available datasets, and experimental results illustrate that VAEAT exhibits robust performance and effective anomaly detection capabilities. The source code of the proposed VAEAT can be available at https://github.com/Du-Team/VAEAT.
Graph anomaly detection aims to identify anomalous occurrences in networks. However, this is more challenging than the traditional anomaly detection problem because anomalies in graphs can manifest ...in three different forms: anomalous nodes, anomalous edges, and anomalous sub-graphs. It is crucial to detect all these anomaly types within a single framework to provide a unified solution to the graph anomaly detection task. The main objective of this study is to propose a model that is capable of detecting all static graph anomalies in a single architecture across various domains. In this paper, we introduce DeGAN (Decomposition-based unified Graph ANomaly detection), a novel framework for unified graph anomaly detection in static networks. DeGAN combines two deep learning concepts with graph decomposition to identify anomalous graph objects: a graph neural network and an adversarial autoencoder. DeGAN is featured with its capability to detect anomalies in a single process, and adopting graph decomposition has improved performance compared to the traditional adversarial learning approach. DeGAN is evaluated with six real-world datasets to demonstrate that our framework can work in multiple domains. Experimental results demonstrate that DeGAN is capable of detecting anomalous nodes, edges, and sub-graphs within a single model. Additionally, the effectiveness of the sub-components of DeGAN has been demonstrated through experimentation. Even though DeGAN is proposed for plain graphs, it can be extended to attributed and dynamic graphs.
•A DML methodology for network intrusion detection.•Triplet networks to deal with data imbalance.•Autoencoders to address the convergence problem of Triplet Networks.•A novel autoencoder-based ...predictive stage.•Experiments with benchmark datasets and various competitors.
Nowadays intrusion detection systems are a mandatory weapon in the war against the ever-increasing amount of network cyber attacks. In this study we illustrate a new intrusion detection method that analyses the flow-based characteristics of the network traffic data. It learns an intrusion detection model by leveraging a deep metric learning methodology that originally combines autoencoders and Triplet networks. In the training stage, two separate autoencoders are trained on historical normal network flows and attacks, respectively. Then a Triplet network is trained to learn the embedding of the feature vector representation of network flows. This embedding moves each flow close to its reconstruction, restored with the autoencoder associated with the same class as the flow, and away from its reconstruction, restored with the autoencoder of the opposite class. The predictive stage assigns each new flow to the class associated with the autoencoder that restores the closest reconstruction of the flow in the embedding space. In this way, the predictive stage takes advantage of the embedding learned in the training stage, achieving a good prediction performance in the detection of new signs of malicious activities in the network traffic. In fact, the proposed methodology leads to better predictive accuracy when compared to competitive intrusion detection architectures on benchmark datasets.
