•A hybrid deep learning model is proposed.•The hybrid model performs very well in hourly streamflow prediction.•Significance of model inputs are investigated by quantifying their contributions to ...predictions.
Process-based streamflow prediction is subjected to large uncertainties in model parameters and parameterizations related to the complex processes involved in streamflow generation. The data-driven models offer efficient alternatives without considering the physical processes, but their applications are limited by non-stationarity existing in observations. In this study, we propose a hybrid model, namely the DIFF-FFNN-LSTM model, to predict hourly streamflow. The model comprises three components, namely the first-order difference (DIFF), feedforward neural network (FFNN), and long short-term memory network (LSTM). When applied to the Andun basin of China, the proposed DIFF-FFNN-LSTM model performs very well in hourly streamflow prediction, with a RMSE of 9.31 m3/s with average streamflow rate of 54 m3/s and a MAE of 3.63 m3/s for all the flood events in the testing period. The comparison with five other machine learning models (of similar complexity or model structure) and four statistical models show superiority of our proposed DIFF-FFNN-LSTM model. The Shapley Additive exPlanations was used to quantify the contribution of each model input to the prediction skill. Streamflow at the previous hour was identified as the most important input, and streamflow generally contribute more than precipitation. Inputs closer to the prediction time do not necessarily have a greater impact on the model prediction. The study highlights the power of the combining of different data-driven methods and the promising prospect of our hybrid model in hydrological predictions.
•The latest applications of deep learning in stock market prediction are presented.•The literature is reviewed with a general workflow for stock market prediction.•The often-ignored implementation ...and reproducibility in other surveys are examined.•The future directions along with the research frontiers are pointed out.
Stock market prediction has been a classical yet challenging problem, with the attention from both economists and computer scientists. With the purpose of building an effective prediction model, both linear and machine learning tools have been explored for the past couple of decades. Lately, deep learning models have been introduced as new frontiers for this topic and the rapid development is too fast to catch up. Hence, our motivation for this survey is to give a latest review of recent works on deep learning models for stock market prediction. We not only category the different data sources, various neural network structures, and common used evaluation metrics, but also the implementation and reproducibility. Our goal is to help the interested researchers to synchronize with the latest progress and also help them to easily reproduce the previous studies as baselines. Based on the summary, we also highlight some future research directions in this topic.
Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often ...viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era.
A reduced basis method based on a physics-informed machine learning framework is developed for efficient reduced-order modeling of parametrized partial differential equations (PDEs). A feedforward ...neural network is used to approximate the mapping from the time-parameter to the reduced coefficients. During the offline stage, the network is trained by minimizing the weighted sum of the residual loss of the reduced-order equations, and the data loss of the labeled reduced coefficients that are obtained via the projection of high-fidelity snapshots onto the reduced space. Such a network is referred to as physics-reinforced neural network (PRNN). As the number of residual points in time-parameter space can be very large, an accurate network – referred to as physics-informed neural network (PINN) – can be trained by minimizing only the residual loss. However, for complex nonlinear problems, the solution of the reduced-order equation is less accurate than the projection of high-fidelity solution onto the reduced space. Therefore, the PRNN trained with the snapshot data is expected to have higher accuracy than the PINN. Numerical results demonstrate that the PRNN is more accurate than the PINN and a purely data-driven neural network for complex problems. During the reduced basis refinement, the PRNN may obtain higher accuracy than the direct reduced-order model based on a Galerkin projection. The online evaluation of PINN/PRNN is orders of magnitude faster than that of the Galerkin reduced-order model.
•Physics-informed machine learning of reduced-order model without requirement of extra high-fidelity snapshots.•A PINN trained by minimizing the residual loss of the reduced-order equation.•A PRNN with improved accuracy obtained by adding the regression loss on the available high-fidelity snapshots data.•Higher accuracy of PRNN on small reduced basis than the direct reduced-order model based on a Galerkin projection.•PRNN as an accurate and efficient reduced-order modeling tool for general nonlinear problems.
A non-intrusive reduced-basis (RB) method is proposed for parametrized unsteady flows. A set of reduced basis functions are extracted from a collection of high-fidelity solutions via a proper ...orthogonal decomposition (POD), and the coefficients of the reduced basis functions are recovered by a feedforward neural network (NN). As a regression model of the RB method for unsteady flows, the neural network approximates the map between the time/parameter value and the projection coefficients of the high-fidelity solution onto the reduced space. The generation of the reduced basis and the training of the NN are accomplished in the offline stage, thus the RB solution of a new time/parameter value can be recovered via direct outputs of the NN in the online stage. Due to its non-intrusive nature, the proposed RB method, referred as the POD-NN, fully decouples the online stage and the high-fidelity scheme, and is thus able to provide fast and reliable solutions of complex unsteady flows. To test this assertion, the POD-NN method is applied to the reduced order modeling (ROM) of the quasi-one dimensional Continuously Variable Resonance Combustor (CVRC) flow. Numerical results demonstrate the efficiency and robustness of the POD-NN method.
•A non-intrusive POD-NN RB method for parametrized unsteady flows.•Feedforward neural network as the regression model.•Time treated as a parameter to account for unsteadiness of the flow.•Capability of POD-NN RB method for ROM of local space/time domain of interest.•Robustness and efficiency of the POD-NN RB method for a complex combustion problem.
•A new model based on LSTM is developed for predicting water table depth.•Only a very simple data pre-processing method is required in our proposed model.•The dropout strategy is adopted to prevent ...over-fitting significantly.•Our model shows superiority over the classic FFNN and the Double-LSTM models.
Predicting water table depth over the long-term in agricultural areas presents great challenges because these areas have complex and heterogeneous hydrogeological characteristics, boundary conditions, and human activities; also, nonlinear interactions occur among these factors. Therefore, a new time series model based on Long Short-Term Memory (LSTM), was developed in this study as an alternative to computationally expensive physical models. The proposed model is composed of an LSTM layer with another fully connected layer on top of it, with a dropout method applied in the first LSTM layer. In this study, the proposed model was applied and evaluated in five sub-areas of Hetao Irrigation District in arid northwestern China using data of 14 years (2000–2013). The proposed model uses monthly water diversion, evaporation, precipitation, temperature, and time as input data to predict water table depth. A simple but effective standardization method was employed to pre-process data to ensure data on the same scale. 14 years of data are separated into two sets: training set (2000–2011) and validation set (2012–2013) in the experiment. As expected, the proposed model achieves higher R2 scores (0.789–0.952) in water table depth prediction, when compared with the results of traditional feed-forward neural network (FFNN), which only reaches relatively low R2 scores (0.004–0.495), proving that the proposed model can preserve and learn previous information well. Furthermore, the validity of the dropout method and the proposed model’s architecture are discussed. Through experimentation, the results show that the dropout method can prevent overfitting significantly. In addition, comparisons between the R2 scores of the proposed model and Double-LSTM model (R2 scores range from 0.170 to 0.864), further prove that the proposed model’s architecture is reasonable and can contribute to a strong learning ability on time series data. Thus, one can conclude that the proposed model can serve as an alternative approach predicting water table depth, especially in areas where hydrogeological data are difficult to obtain.
Robot manipulators are playing increasingly significant roles in scientific researches and engineering applications in recent years. Using manipulators to save labors and increase accuracies are ...becoming common practices in industry. Neural networks, which feature high-speed parallel distributed processing, and can be readily implemented by hardware, have been recognized as a powerful tool for real-time processing and successfully applied widely in various control systems. Particularly, using neural networks for the control of robot manipulators have attracted much attention and various related schemes and methods have been proposed and investigated. In this paper, we make a review of research progress about controlling manipulators by means of neural networks. The problem foundation of manipulator control and the theoretical ideas on using neural network to solve this problem are first analyzed and then the latest progresses on this topic in recent years are described and reviewed in detail. Finally, toward practical applications, some potential directions possibly deserving investigation in controlling manipulators by neural networks are pointed out and discussed.
Display omitted
•ANN applications on solving solid waste related issues in last decade are reviewed.•Studies are classified into macroscale, mesoscale, meso–microscale and microscale.•Various ...configurations on ANN framework and performance evaluation are compared.•Preferable settings and promising application in waste related studies are discussed.
Artificial neural networks (ANNs) have recently attracted significant attention in environmental areas because of their great self-learning capability and good accuracy in mapping complex nonlinear relationships. These properties of ANNs benefit their application in solving different solid waste-related issues. However, the configurations, including ANN framework, algorithm, data set partition, input parameters, hidden layer, and performance evaluation, vary and have not reached a consensus among relevant studies. To address the current state of the art of ANN application in the solid waste field and identify the commonalities of ANNs, this critical review was conducted by focusing on a modeling perspective and using 177 relevant papers published over the last decade (2010–2020). We classified the reviewed studies into four categories in terms of research scales. ANNs were found to be applied widely in waste generation and technological parameter prediction and proven effective in solving meso–microscale and microscale issues, including waste conversion, emissions, and microbial and dynamic processes. Given the difficulty of data collection in many solid waste-related issues, most studies included a data size of 101–150. For mathematical optimization, dividing the data into training–validation–test sets is preferable, and the training set is supposed to account for ~70%. A single hidden layer is usually sufficient, and the optimal numbers of hidden layer nodes most likely range from 4 to 20. This review is supposed to contribute basic and comprehensive knowledge to the researchers in general waste management and specialized ANN study on solid waste-related issues.
Handwritten Chinese text recognition based on over-segmentation and path search integrating multiple contexts has been demonstrated successful, wherein the language model (LM) and character shape ...models play important roles. Although back-off N-gram LMs (BLMs) have been used dominantly for decades, they suffer from the data sparseness problem, especially for high-order LMs. Recently, neural network LMs (NNLMs) have been applied to handwriting recognition with superiority to BLMs. With the aim of improving Chinese handwriting recognition, this paper evaluates the effects of two types of character-level NNLMs, namely, feedforward neural network LMs (FNNLMs) and recurrent neural network LMs (RNNLMs). Both FNNLMs and RNNLMs are also combined with BLMs to construct hybrid LMs. For fair comparison with BLMs and a state-of-the-art system, we evaluate in a system with the same character over-segmentation and classification techniques as before, and compare various LMs using a small text corpus used before. Experimental results on the Chinese handwriting database CASIA-HWDB validate that NNLMs improve the recognition performance, and hybrid RNNLMs outperform the other LMs. To report a new benchmark, we also evaluate selected LMs on a large corpus, and replace the baseline character classifier, over-segmentation, and geometric context models with convolutional neural network (CNN) based models. The performance on both the CASIA-HWDB and the ICDAR-2013 competition dataset are improved significantly. On the CASIA-HWDB test set, the character-level accurate rate (AR) and correct rate (CR) achieve 95.88% and 95.95%, respectively.
•We evaluate comprehensively neural network language models (NNLMs) and hybrid NNLMs in handwritten Chinese text recognition.•We apply CNNs to over-segmentation and geometric context modeling in addition to character recognition.•By training NNLMs on large corpus and integrating CNN shape models, we achieve new state-of-the-art performance on standard datasets.•We analyze the upper bound of performance of the text recognition system by calculating the lattice error rate.
Financial time-series forecasting has long been a challenging problem because of the inherently noisy and stochastic nature of the market. In the high-frequency trading, forecasting for trading ...purposes is even a more challenging task, since an automated inference system is required to be both accurate and fast. In this paper, we propose a neural network layer architecture that incorporates the idea of bilinear projection as well as an attention mechanism that enables the layer to detect and focus on crucial temporal information. The resulting network is highly interpretable, given its ability to highlight the importance and contribution of each temporal instance, thus allowing further analysis on the time instances of interest. Our experiments in a large-scale limit order book data set show that a two-hidden-layer network utilizing our proposed layer outperforms by a large margin all existing state-of-the-art results coming from much deeper architectures while requiring far fewer computations.
PDF
