•We employed adaptive activation functions in deep and physics-informed neural networks.•The proposed method is very simple and it is shown to accelerate convergence in neural networks.•In ...particular, we approximate various nonlinear functions using deep neural networks.•Physics-informed neural networks are employed to solve both forward and inverse problems of PDEs.•We also solved standard deep learning benchmarks problems and theoretically proved convergence results.
We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima. Furthermore, the proposed adaptive activation functions are shown to accelerate the minimization process of the loss values in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion datasets with and without data augmentation.
In this work we investigate the possibility of using physics-informed neural networks (PINNs) to approximate the Euler equations that model high-speed aerodynamic flows. In particular, we solve both ...the forward and inverse problems in one-dimensional and two-dimensional domains. For the forward problem, we utilize the Euler equations and the initial/boundary conditions to formulate the loss function, and solve the one-dimensional Euler equations with smooth solutions and with solutions that have a contact discontinuity as well as a two-dimensional oblique shock wave problem. We demonstrate that we can capture the solutions with only a few scattered points clustered randomly around the discontinuities. For the inverse problem, motivated by mimicking the Schlieren photography experimental technique used traditionally in high-speed aerodynamics, we use the data on density gradient ∇ρ(x,t), the pressure p(x∗,t) at a specified point x=x∗ as well as the conservation laws to infer all states of interest (density, velocity and pressure fields). We present illustrative benchmark examples for both the problem with smooth solutions and Riemann problems (Sod and Lax problems) with PINNs, demonstrating that all inferred states are in good agreement with the reference solutions. Moreover, we show that the choice of the position of the point x∗ plays an important role in the learning process. In particular, for the problem with smooth solutions we can randomly choose the position of the point x∗ from the computational domain, while for the Sod or Lax problem, we have to choose the position of the point x∗ from the domain between the initial discontinuous point and the shock position of the final time. We also solve the inverse problem by combining the aforementioned data and the Euler equations in characteristic form, showing that the results obtained by using the Euler equations in characteristic form are better than that obtained by using the Euler equations in conservative form. Furthermore, we consider another type of inverse problem, specifically, we employ PINNs to learn the value of the parameter γ in the equation of state for the parameterized two-dimensional oblique wave problem by using the given data of the density, velocity and the pressure, and we identify the parameter γ accurately. Taken together, our results demonstrate that in the current form, where the conservation laws are imposed at random points, PINNs are not as accurate as traditional numerical methods for forward problems but they are superior for inverse problems that cannot even be solved with standard techniques.
We propose a conservative physics-informed neural network (cPINN) on discrete domains for nonlinear conservation laws. Here, the term discrete domain represents the discrete sub-domains obtained ...after division of the computational domain, where PINN is applied and the conservation property of cPINN is obtained by enforcing the flux continuity in the strong form along the sub-domain interfaces. In case of hyperbolic conservation laws, the convective flux contributes at the interfaces, whereas in case of viscous conservation laws, both convective and diffusive fluxes contribute. Apart from the flux continuity condition, an average solution (given by two different neural networks) is also enforced at the common interface between two sub-domains. One can also employ a deep neural network in the domain, where the solution may have complex structure, whereas a shallow neural network can be used in the sub-domains with relatively simple and smooth solutions. Another advantage of the proposed method is the additional freedom it gives in terms of the choice of optimization algorithm and the various training parameters like residual points, activation function, width and depth of the network etc. Various forms of errors involved in cPINN such as optimization, generalization and approximation errors and their sources are discussed briefly. In cPINN, locally adaptive activation functions are used, hence training the model faster compared to its fixed counterparts. Both, forward and inverse problems are solved using the proposed method. Various test cases ranging from scalar nonlinear conservation laws like Burgers, Korteweg–de Vries (KdV) equations to systems of conservation laws, like compressible Euler equations are solved. The lid-driven cavity test case governed by incompressible Navier–Stokes equation is also solved and the results are compared against a benchmark solution. The proposed method enjoys the property of domain decomposition with separate neural networks in each sub-domain, and it efficiently lends itself to parallelized computation, where each sub-domain can be assigned to a different computational node.
•Domain decomposition technique is proposed for PINNs with tailored neural network in each sub-domain for solving conservation laws.•Due to presence of multiple neural networks, the representation capacity of the proposed cPINN method increases.•Based on prior knowledge of the solution regularity in each sub-domain, the hyper-parameter set of corresponding PINN can be properly adjusted.•The partial independence of individual PINNs in decomposed domains can be further employed to implement cPINN in a parallelized algorithm.
•Construction and implementation of new domain-decomposition based parallel algorithm is proposed for cPINNs and XPINNs methods.•The proposed algorithm adds another dimension of parallelism in SciML ...primarily driven by data and model parallelism.•The proposed algorithm is shown its scaling for CPU and CPU+GPU architectures.
We develop a distributed framework for the physics-informed neural networks (PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and extended PINNs (XPINNs), which employ domain decomposition in space and in time-space, respectively. This domain decomposition endows cPINNs and XPINNs with several advantages over the vanilla PINNs, such as parallelization capacity, large representation capacity, efficient hyperparameter tuning, and is particularly effective for multi-scale and multi-physics problems. Here, we present a parallel algorithm for cPINNs and XPINNs constructed with a hybrid programming model described by MPI + X, where X ∈{CPUs,GPUs}. The main advantage of cPINN and XPINN over the more classical data and model parallel approaches is the flexibility of optimizing all hyperparameters of each neural network separately in each subdomain. We compare the performance of distributed cPINNs and XPINNs for various forward problems, using both weak and strong scalings. Our results indicate that for space domain decomposition, cPINNs are more efficient in terms of communication cost but XPINNs provide greater flexibility as they can also handle time-domain decomposition for any differential equations, and can deal with any arbitrarily shaped complex subdomains. To this end, we also present an application of the parallel XPINN method for solving an inverse diffusion problem with variable conductivity on the United States map, using ten regions as subdomains.
Accurate solutions to inverse supersonic compressible flow problems are often required for designing specialized aerospace vehicles. In particular, we consider the problem where we have data ...available for density gradients from Schlieren photography as well as data at the inflow and part of the wall boundaries. These inverse problems are notoriously difficult, and traditional methods may not be adequate to solve such ill-posed inverse problems. To this end, we employ the physics-informed neural networks (PINNs) and its extended version, extended PINNs (XPINNs), where domain decomposition allows to deploy locally powerful neural networks in each subdomain, which can provide additional expressivity in subdomains, where a complex solution is expected. Apart from the governing compressible Euler equations, we also enforce the entropy conditions in order to obtain viscosity solutions. Moreover, we enforce positivity conditions on density and pressure. We consider inverse problems involving two-dimensional expansion waves, two-dimensional oblique and bow shock waves. We compare solutions obtained by PINNs and XPINNs and invoke some theoretical results that can be used to decide on the generalization errors of the two methods.
We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed ...neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope-based
term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix-vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.
Abstract
We prove rigorous bounds on the errors resulting from the approximation of the incompressible Navier–Stokes equations with (extended) physics-informed neural networks. We show that the ...underlying partial differential equation residual can be made arbitrarily small for tanh neural networks with two hidden layers. Moreover, the total error can be estimated in terms of the training error, network size and number of quadrature points. The theory is illustrated with numerical experiments.
One- and two-dimensional sine-Gordon equation in non-homogeneous media is considered. Sine-Gordon equation exhibits soliton-like solution whose existence and behaviour in non-homogeneous media is ...studied. Various non-homogeneous media are discussed which include nonlinear smooth as well as discontinuous density variations. Time-dependent continuous and discontinuous density variations which can model geodynamical processes like lithospheric deformation, fault dynamics, etc. are also discussed. In one dimension, dynamics of kink which are topological soliton is studied under such density variations, whereas in two dimensions, circular-ring soliton dynamics has been scrutinized. The governing sine-Gordon equation is solved numerically using higher order Legendre polynomial-based spectral element method. Spectral stability analysis of the numerical scheme shows the strong dependence of a critical time step not only on density value but also on the nature of the distribution. It is shown that for smooth density variation the critical time step reduces drastically which makes explicit scheme very expensive. To reduce the computational cost, unconditionally stable fully implicit spectral element scheme is proposed which is also energy conservative.
In this paper we proposed the kinetic framework based fifth-order adaptive finite difference WENO schemes abbreviated as WENO-AO-K schemes to solve the compressible Euler equations, which are ...quasi-linear hyperbolic equations that can admit discontinuous solutions like shock and contact waves. The formulation of the proposed schemes is based on the kinetic theory where one can recover the Euler equations by applying a suitable moment method strategy to the Boltzmann equation. The kinetic flux vector splitting strategy is used in WENO-AO framework, which produces the computationally expensive error and exponential functions. Thus, to reduce the computational cost, a physically more relevant peculiar velocity based splitting strategy is used, which is more efficient than the kinetic flux vector splitting. High order of accuracy in time is achieved using the third-order total variation diminishing Runge–Kutta (TVD-RK) scheme. Several one- and two-dimensional test cases are solved for the compressible Euler equations using the proposed fifth-order WENO-AO-K schemes and the results are compared with conventional WENO-AO scheme. Proposed schemes capture the complex flow features in a smooth region accurately, and discontinuity is also well resolved. Error analysis of the proposed schemes shows optimal convergence rates in various norms.
In this work a new finite element based Method of Relaxed Streamline Upwinding is proposed to solve hyperbolic conservation laws. Formulation of the proposed scheme is based on relaxation system ...which replaces hyperbolic conservation laws by semi-linear system with stiff source term also called as relaxation term. The advantage of the semi-linear system is that the nonlinearity in the convection term is pushed towards the source term on right hand side which can be handled with ease. Six symmetric discrete velocity models are introduced in two dimensions which symmetrically spread foot of the characteristics in all four quadrants thereby taking information symmetrically from all directions. Proposed scheme gives exact diffusion vectors which are very simple. Moreover, the formulation is easily extendable from scalar to vector conservation laws. Various test cases are solved for Burgers equation (with convex and non-convex flux functions), Euler equations and shallow water equations in one and two dimensions which demonstrate the robustness and accuracy of the proposed scheme. New test cases are proposed for Burgers equation, Euler and shallow water equations. Exact solution is given for two-dimensional Burgers test case which involves normal discontinuity and series of oblique discontinuities. Error analysis of the proposed scheme shows optimal convergence rate. Moreover, spectral stability analysis gives implicit expression of critical time step.
•In this paper relaxation system based new stabilized finite element scheme is proposed for hyperbolic conservation laws.•Six symmetric discrete velocity models based on orthogonal velocity method are proposed for two-dimensional problems.•Proposed scheme gives simple diffusion terms required for stabilization of the numerical scheme.•Proposed scheme can be directly extended from scalar to vector conservation laws.•Various test cases of Burgers equation, Euler and shallow water equations are solved in one and two dimensions.
PDF
