Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains. Deep neural network architectures and computational issues have been ...well studied in machine learning. But there lacks a theoretical foundation for understanding the approximation or generalization ability of deep learning methods generated by the network architectures such as deep convolutional neural networks. Here we show that a deep convolutional neural network (CNN) is universal, meaning that it can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. This answers an open question in learning theory. Our quantitative estimate, given tightly in terms of the number of free parameters to be computed, verifies the efficiency of deep CNNs in dealing with large dimensional data. Our study also demonstrates the role of convolutions in deep CNNs.
Numerical simulation of multiphase flow in porous media is essential for many geoscience applications. Machine learning models trained with numerical simulation data can provide a faster alternative ...to traditional simulators. Here we present U-FNO, a novel neural network architecture for solving multiphase flow problems with superior accuracy, speed, and data efficiency. U-FNO is designed based on the newly proposed Fourier neural operator (FNO), which has shown excellent performance in single-phase flows. We extend the FNO-based architecture to a highly complex CO2-water multiphase problem with wide ranges of permeability and porosity heterogeneity, anisotropy, reservoir conditions, injection configurations, flow rates, and multiphase flow properties. The U-FNO architecture is more accurate in gas saturation and pressure buildup predictions than the original FNO and a state-of-the-art convolutional neural network (CNN) benchmark. Meanwhile, it has superior data utilization efficiency, requiring only a third of the training data to achieve the equivalent accuracy as CNN. U-FNO provides superior performance in highly heterogeneous geological formations and critically important applications such as gas saturation and pressure buildup “fronts” determination. The trained model can serve as a general-purpose alternative to routine numerical simulations of 2D-radial CO2 injection problems with significant speed-ups than traditional simulators.
•U-FNO model for multiphase flow designed based on Fourier neural operator.•Data-efficient multiphase flow predictions for gas saturation and pressure buildup.•Results are significantly faster and more accurate than state-of-the-art CNNs.
Deep learning (DL) architecture, which exploits multiple hidden layers to learn hierarchical representations automatically from massive input data, presents a promising tool for characterizing fault ...conditions. This paper proposes a DL-based multi-signal fault diagnosis method that leverages the powerful feature learning ability of a convolutional neural network (CNN) in images. The proposed deep model is able to learn from multiple types of sensor signals simultaneously so that it can achieve robust performance and finally realize accurate induction motor fault recognition. First, the acquired sensor signals are converted to time-frequency distribution (TFD) by wavelet transform. Then, a deep CNN is applied to learning discriminative representations from the TFD images. Since then, a fully connected layer in deep architecture gives the prediction of induction motor condition based on learned features. In order to verify the effectiveness of the designed deep model, experiments are carried out on a machine fault simulator where both vibration and current signals are analyzed. Experimental results indicate that the proposed method outperforms traditional fault diagnosis methods, hence, demonstrating effectiveness in induction motor application. Compared with conventional methods that rely on delicate features extracted by experienced experts, the proposed deep model is able to automatically learn and select suitable features that contribute to accurate fault diagnosis. Compared with single-signal input, the multi-signal model has more accurate and stable performance and overcomes the overfitting problem to some degree.
Change detection based on remote sensing (RS) images has a wide range of applications in many fields. However, many existing approaches for detecting changes in RS images with complex land covers ...still have room for improvement. In this article, a high-resolution RS image change detection approach based on a deep feature difference convolutional neural network (CNN) is proposed. This approach uses a CNN to learn the deep features from RS images and then uses transfer learning to compose a two-channel network with shared weight to generate a multiscale and multidepth feature difference map for change detection. The network is trained by a change magnitude guided loss function proposed in this article and needs to train with only a few pixel-level samples to generate change magnitude maps, which can help to remove some of the pseudochanges. Finally, the binary change map can be obtained by a threshold. The approach is tested on several data sets from different sensors, including WorldView-3, QuickBird, and Ziyuan-3. The experimental results show that the proposed approach achieves better performance compared with other classic approaches and has fewer missed detections and false alarms, which proves that the proposed approach has strong robustness and generalization ability.
Accurate short-term passenger demand prediction contributes to the coordination of traffic supply and demand. This paper proposes an end-to-end multi-task learning temporal convolutional neural ...network (MTL-TCNN) to predict the short-term passenger demand in a multi-zone level. Along with a feature selector named spatiotemporal dynamic time warping (ST-DTW) algorithm, this proposed MTL-TCNN is quite qualified for the multi-task prediction problem with the consideration of spatiotemporal correlations. Then, based on the car-calling demand data from Didi Chuxing, Chengdu, China, and taxi demand data from the New York City, the numerical results show that the MTL-TCNN outperforms both classic methods (i.e., historical average (HA), v -support vector machine (v -SVM), and XGBoost) and the state-of-the-art deep learning approaches e.g., long short-term memory (LSTM) and convolutional LSTM (ConvLSTM) in both the single task learning (STL) and multi-task learning (MTL) scenarios. In summary, the proposed MTL-TCNN with the ST-DTW algorithm is a promising method for short-term passenger demand prediction in a multi-zone level.
Scene classification of remote sensing images has drawn great attention because of its wide applications. In this paper, with the guidance of the human visual system (HVS), we explore the attention ...mechanism and propose a novel end-to-end attention recurrent convolutional network (ARCNet) for scene classification. It can learn to focus selectively on some key regions or locations and just process them at high-level features, thereby discarding the noncritical information and promoting the classification performance. The contributions of this paper are threefold. First, we design a novel recurrent attention structure to squeeze high-level semantic and spatial features into several simplex vectors for the reduction of learning parameters. Second, an end-to-end network named ARCNet is proposed to adaptively select a series of attention regions and then to generate powerful predictions by learning to process them sequentially. Third, we construct a new data set named OPTIMAL-31, which contains more categories than popular data sets and gives researchers an extra platform to validate their algorithms. The experimental results demonstrate that our model makes great promotion in comparison with the state-of-the-art approaches.
Fault diagnosis, which identifies the root cause of the observed out-of-control status, is essential to counteracting or eliminating faults in industrial processes. Many conventional data-driven ...fault diagnosis methods ignore the fault tendency of abnormal samples, and they need a complete retraining process to include the newly collected abnormal samples or fault classes. In this article, a broad convolutional neural network (BCNN) is designed with incremental learning capability for solving the aforementioned issues. The proposed method combines several consecutive samples as a data matrix, and it then extracts both fault tendency and nonlinear structure from the obtained data matrix by using convolutional operation. After that, the weights in fully connected layers can be trained based on the obtained features and their corresponding fault labels. Because of the architecture of this network, the diagnosis performance of the BCNN model can be improved by adding newly generated additional features. Finally, the incremental learning capability of the proposed method is also designed, so that the BCNN model can update itself to include new coming abnormal samples and fault classes. The proposed method is applied both to a simulated process and a real industrial process. Experimental results illustrate that it can better capture the characteristics of the fault process, and effectively update diagnosis model to include new coming abnormal samples, and fault classes.
Change detection is one of the central problems in earth observation and was extensively investigated over recent decades. In this paper, we propose a novel recurrent convolutional neural network ...(ReCNN) architecture, which is trained to learn a joint spectral-spatial-temporal feature representation in a unified framework for change detection in multispectral images. To this end, we bring together a convolutional neural network and a recurrent neural network into one end-to-end network. The former is able to generate rich spectral-spatial feature representations, while the latter effectively analyzes temporal dependence in bitemporal images. In comparison with previous approaches to change detection, the proposed network architecture possesses three distinctive properties: 1) it is end-to-end trainable, in contrast to most existing methods whose components are separately trained or computed; 2) it naturally harnesses spatial information that has been proven to be beneficial to change detection task; and 3) it is capable of adaptively learning the temporal dependence between multitemporal images, unlike most of the algorithms that use fairly simple operation like image differencing or stacking. As far as we know, this is the first time that a recurrent convolutional network architecture has been proposed for multitemporal remote sensing image analysis. The proposed network is validated on real multispectral data sets. Both visual and quantitative analyses of the experimental results demonstrate competitive performance in the proposed mode.
•We give an overview of the basic components of CNN.•We discuss the improvements of CNN on different aspects, namely, layer design, activation function, loss function, regularization, optimization ...and fast computation.•We introduce the applications of CNN on various tasks, including image classification, object detection, object tracking, pose estimation, text detection, visual saliency detection, action recognition, scene labeling, speech and natural language processing.•We discuss the challenges in CNN and give several future research directions.
In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Among different types of deep neural networks, convolutional neural networks have been most extensively studied. Leveraging on the rapid growth in the amount of the annotated data and the great improvements in the strengths of graphics processor units, the research on convolutional neural networks has been emerged swiftly and achieved state-of-the-art results on various tasks. In this paper, we provide a broad survey of the recent advances in convolutional neural networks. We detailize the improvements of CNN on different aspects, including layer design, activation function, loss function, regularization, optimization and fast computation. Besides, we also introduce various applications of convolutional neural networks in computer vision, speech and natural language processing.
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based ...method for this task is proposed, by learning a nonlinear end-to-end mapping between the noisy and clean HSIs with a combined spatial-spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multiscale feature extraction and multilevel feature representation are, respectively, employed to capture both the multiscale spatial-spectral feature and fuse different feature representations for the final restoration. The simulated and real-data experiments demonstrate that the proposed HSID-CNN outperforms many of the mainstream methods in both the quantitative evaluation indexes, visual effects, and HSI classification accuracy.
PDF
