•Evidence Lower Bound on incomplete datasets, computed only on the observed data, regardless of the pattern of missing data.•Generative model that handles mixed numerical and nominal likelihood ...models, parametrized using deep neural networks (DNNs).•Stable recognition model that handles incomplete datasets without increasing its complexity or promoting overfitting.•Data-normalization input/output layer prevents a few dimensions of the data dominating the training of the VAE, improving the training convergence.•Comparison with state-of-the-art methods on six datasets for both missing data imputation and predictive tasks.
Variational autoencoders (VAEs), as well as other generative models, have been shown to be efficient and accurate for capturing the latent structure of vast amounts of complex high-dimensional data. However, existing VAEs can still not directly handle data that are heterogenous (mixed continuous and discrete) or incomplete (with missing data at random), which is indeed common in real-world applications.
In this paper, we propose a general framework to design VAEs suitable for fitting incomplete heterogenous data. The proposed HI-VAE includes likelihood models for real-valued, positive real valued, interval, categorical, ordinal and count data, and allows accurate estimation (and potentially imputation) of missing data. Furthermore, HI-VAE presents competitive predictive performance in supervised tasks, outperforming supervised models when trained on incomplete data.
Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of ...which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated, while reviewing current state-of-the-art advances and implementations.
Concurrent process-quality monitoring helps discover quality-relevant process anomalies and quality-irrelevant process anomalies. It especially works well in chemical plants with faults that cause ...quality problems. Traditional monitoring strategies are limitedly applied in chemical plants because quality targets in training data are insufficient. It is hard for inflexible models to fully capture the strongly nonlinear process-quality correlations. Also, deterministic models are mapped from process variables to qualities without any consideration of uncertainties. Simultaneously, a slow sampling rate for quality variables is ubiquitous in chemical plants since a product quality test is often time-consuming and expensive. Motivated by these limitations, this paper proposes a new concurrent process-quality monitoring scheme based on a probabilistic generative deep learning model developed from variational autoencoder. The supervised model is firstly developed and then the semi-supervised version is extended to solve the issue of missing targets. Especially, the semi-supervised learning algorithm is accomplished with an optimal parameter estimation in the light of maximum likelihood principle and no any hyperparameters are introduced. Two case studies validate that the proposed method effectively outperforms the other comparative methods in concurrent process-quality monitoring.
•A probabilistic deep network for learning correlations between process and quality.•The semi-supervised network dealing with slow-sampling quality variables.•Unified framework integrating supervised and semi-supervised network training.•Application of concurrent process-quality monitoring.
Variational autoencoders (VAEs) are a class of effective deep generative models, with the objective to approximate the true, but unknown data distribution. VAEs make use of latent variables to ...capture high-level semantics so as to reconstruct the data well with the help of informative latent variables. Yet, training VAEs tends to suffer from posterior collapse, when the decoder is parameterized by an autoregressive model for sequence generation. VAEs can be further enhanced by introducing multiple layers of latent variables, but the posterior collapse issue hinders the adoption of such hierarchical VAEs in real-world applications. In this paper, we introduce InfoMaxHVAE, which integrates mutual information estimated via neural networks into hierarchical VAEs to alleviate posterior collapse, when powerful autoregressive models are used for modeling sequences. Experimental results on a number of text and image datasets show that InfoMaxHVAE can outperform the state-of-the-art baselines and exhibits less posterior collapse. We further show that InfoMaxHVAE can shape a coarse-to-fine hierarchical organization of the latent space.
We propose the first stochastic framework to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Existing RGB-D saliency detection models treat this task as a ...point estimation problem by predicting a single saliency map following a deterministic learning pipeline. We argue that, however, the deterministic solution is relatively ill-posed. Inspired by the saliency data labeling process, we propose a generative architecture to achieve probabilistic RGB-D saliency detection which utilizes a latent variable to model the labeling variations. Our framework includes two main models: 1) a generator model, which maps the input image and latent variable to stochastic saliency prediction, and 2) an inference model, which gradually updates the latent variable by sampling it from the true or approximate posterior distribution. The generator model is an encoder-decoder saliency network. To infer the latent variable, we introduce two different solutions: i) a Conditional Variational Auto-encoder with an extra encoder to approximate the posterior distribution of the latent variable; and ii) an Alternating Back-Propagation technique, which directly samples the latent variable from the true posterior distribution. Qualitative and quantitative results on six challenging RGB-D benchmark datasets show our approach's superior performance in learning the distribution of saliency maps. The source code is publicly available via our project page: https://github.com/JingZhang617/UCNet .
Variational autoencoders (VAEs) have shown promising potential as artificial neural networks (NN) for developing reduced-order models (ROMs) in the context of turbulent flows. In this study, we ...propose a method that combines β-VAEs for modal decomposition and transformer neural networks for temporal-dynamics prediction in the latent space to develop ROMs. We apply our method to an existing database of a turbulent flow around a wall-mounted square cylinder obtained by direct numerical simulation (DNS). A parametric study is performed to investigate the effects of the hyperparameters of the proposed β-VAEs and determine the optimal values. For the first time, we incorporate the consideration of the complexity of architecture into our studies, providing new insights into hyperparameter selection for β-VAEs, which remains a challenging problem for optimising model performance. Results regarding the influence of the different hyperparameters and guidelines to design these architectures are reported. Our optimal model achieves a reconstruction accuracy of 97.18% of the entire dataset using only ten modes. Subsequently, we employ the transformer models to identify latent-space temporal dynamics learned by the optimal β-VAE model and build ROMs to predict instantaneous fields. The resulting model achieves promising accuracy in temporal-dynamics predictions and yields energy reconstruction levels of 96.5% and 83% for a field 25 and 50 steps into the future, respectively, showcasing the potential of the transformer in predicting the temporal dynamics. Overall, the proposed method has potential applications in advanced flow control and fundamental studies of complex turbulent flows.
This article utilizes variational autoencoder (VAE) and spread spectrum time domain reflectometry (SSTDR) to detect, isolate, and characterize anomalous data (or faults) in a photovoltaic (PV) array. ...The goal is to learn the distribution of non-faulty input signals, inspect the reconstruction error of test signals, flag anomalies, and then locate or characterize the anomalous data using a predicted baseline rather than a fixed baseline that might be too rigid. The use of VAE handles imbalanced data better than other methods used for classification of PV faults because of its unsupervised nature. Here, we consider only disconnects in this work, and our results show an overall accuracy of 96% for detecting true negatives (non-faulty data), a 99% true positive rate of detecting anomalies, 0.997 area under the ROC curve, 0.99 area under the precision-recall curve, and a maximum percentage absolute relative error of 0.40% in locating the faults on a 5-panel setup with a 59.13 m leader cable.
A ventilated acoustic resonator (VAR), a type of acoustic metamaterial (AM) has emerged as a promising solution for mitigating urban noise pollution and traffic noise which simultaneously require ...ventilation. However, due to the high nonlinearity, the inverse design of complex VAR is intractable with analytical methods. Deep learning-based inverse design methods are gaining prominence as an alternative to analytical methods but still exhibit significant challenges: limited design flexibility in parameter-based approaches and the deterioration of essential shapes for sound attenuation performance in pixel image-based approaches. To address these challenges, we propose an inverse design framework of ultra-broadband non-parametric VAR through a genetic algorithm (GA) optimization-based latent space exploration strategy. The GA-based exploration on the dimension-reduced latent space of a conditional variational autoencoder (CVAE) enables the generation of the ultra-broadband non-parametric VAR preserving essential shape for sound attenuation with reduced computational costs. The GA-optimized non-parametric VARs show an average 28.76% bandwidth increase compared with the training dataset and, also demonstrate a considerably wider bandwidth compared to the parameter-based optimization methods, which expands the limit of the sound attenuation performance. Our novel approach paves the way for the optimization of complex mechanical structures.
PDF
