In this brief, a reinforcement learning-based control approach for nonlinear systems is presented. The proposed control approach offers a design scheme of the adjustable policy learning rate (APLR) ...to reduce the influence imposed by negative or large advantages, which improves the learning stability of the proximal policy optimization (PPO) algorithm. Besides, this brief puts forward a Lyapunov-fuzzy reward system to further promote the learning efficiency. In addition, the proposed control approach absorbs the Lyapunov stability concept into the design of the Lyapunov reward system and a particular fuzzy reward system is set up using the knowledge of the cart-pole inverted pendulum and fuzzy inference system (FIS). The merits of the proposed approach are validated by simulation examples.
This paper presents an adaptive neuro-fuzzy PID controller based on twin delayed deep deterministic policy gradient (TD3) algorithm for nonlinear systems. In this approach, the observation of the ...environment is embedded with information of a multiple input single output (MISO) fuzzy inference system (FIS) and have a specially defined fuzzy PID controller in neural network (NN) formation acting as the actor in the TD3 algorithm, which achieves automatic tuning of gains of fuzzy PID controller. From the control perspective, the controller combines the merits of both FIS and PID controller and utilizes reinforcement learning algorithm for optimizing parameters. From the reinforcement learning point of view, embedding the prior knowledge into the fuzzy PID controller incorporated in the actor network helps reduce the learning difficulty in the training process. The proposed method was tested on the cart-pole system in simulation environment with comparison of a linear PID controller, which demonstrates the robustness and generalization of the proposed approach.
This paper proposes an adaptive super-twisting decoupled terminal sliding mode control technique for a class of fourth-order systems. The adaptive-tuning law eliminates the requirement of the ...knowledge about the upper bounds of external perturbations. Using the proposed control procedure, the state variables of cart-pole system are converged to decoupled terminal sliding surfaces and their equilibrium points in the finite time. Moreover, via the super-twisting algorithm, the chattering phenomenon is avoided without affecting the control performance. The numerical results demonstrate the high stabilization accuracy and lower performance indices values of the suggested method over the other ones. The simulation results on the cart-pole system as well as experimental validations demonstrate that the proposed control technique exhibits a reasonable performance in comparison with the other methods.
•Design of adaptive super-twisting decoupled terminal sliding mode control for stabilization of fourth-order systems.•Suggestion of a suitable adaptive parameter-tuning law to dominate perturbations without knowledge of upper bounds.•Establishment of chattering-free robust performance and finite-time convergence for both subsystems.•Verification of the proposed technique by demonstrative simulation results and experimental assessments.
This paper describes the learning and control capabilities of a biologically constrained bottom-up model of the mammalian cerebellum. Results are presented from six tasks: 1) eyelid conditioning; 2) ...pendulum balancing; 3) proportional-integral-derivative control; 4) robot balancing; 5) pattern recognition; and 6) MNIST handwritten digit recognition. These tasks span several paradigms of machine learning, including supervised learning, reinforcement learning, control, and pattern recognition. Results over these six domains indicate that the cerebellar simulation is capable of robustly identifying static input patterns even when randomized across the sensory apparatus. This capability allows the simulated cerebellum to perform several different supervised learning and control tasks. On the other hand, both reinforcement learning and temporal pattern recognition prove problematic due to the delayed nature of error signals and the simulator's inability to solve the credit assignment problem. These results are consistent with previous findings which hypothesize that in the human brain, the basal ganglia is responsible for reinforcement learning, while the cerebellum handles supervised learning.
Several factors combine to make it feasible to build computer simulations of the cerebellum and to test them in biologically realistic ways. These simulations can be used to help understand the ...computational contributions of various cerebellar components, including the relevance of the enormous number of neurons in the granule cell layer. In previous work we have used a simulation containing 12000 granule cells to develop new predictions and to account for various aspects of eyelid conditioning, a form of motor learning mediated by the cerebellum. Here we demonstrate the feasibility of scaling up this simulation to over one million granule cells using parallel graphics processing unit (GPU) technology. We observe that this increase in number of granule cells requires only twice the execution time of the smaller simulation on the GPU. We demonstrate that this simulation, like its smaller predecessor, can emulate certain basic features of conditioned eyelid responses, with a slight improvement in performance in one measure. We also use this simulation to examine the generality of the computation properties that we have derived from studying eyelid conditioning. We demonstrate that this scaled up simulation can learn a high level of performance in a classic machine learning task, the cart–pole balancing task. These results suggest that this parallel GPU technology can be used to build very large-scale simulations whose connectivity ratios match those of the real cerebellum and that these simulations can be used guide future studies on cerebellar mediated tasks and on machine learning problems.
► One million cell simulation using GPGPU for reasonable execution time. ► Tool to begin address the computational role of large number of granule cells. ► Examined generality of simulation that is derived from eyelid conditioning.
Abstract
Recent advances in classical reinforcement learning (RL) and quantum computation point to a promising direction for performing RL on a quantum computer. However, potential applications in ...quantum RL are limited by the number of qubits available in modern quantum devices. Here, we present two frameworks for deep quantum RL tasks using gradient-free evolutionary optimization. First, we apply the amplitude encoding scheme to the Cart-Pole problem, where we demonstrate the quantum advantage of parameter saving using amplitude encoding. Second, we propose a hybrid framework where the quantum RL agents are equipped with a hybrid tensor network-variational quantum circuit (TN-VQC) architecture to handle inputs of dimensions exceeding the number of qubits. This allows us to perform quantum RL in the MiniGrid environment with 147-dimensional inputs. The hybrid TN-VQC architecture provides a natural way to perform efficient compression of the input dimension, enabling further quantum RL applications on noisy intermediate-scale quantum devices.
In this study, we investigate model structures for neural ODEs to improve the data efficiency in learning the dynamics of control systems. We introduce two model structures and compare them with a ...typical baseline structure. The first structure considers the relationship between the coordinates and velocities of the control system, while the second structure adds linearity with respect to the control term to the first structure. Both of these structures can be easily implemented without requiring additional computation. In numerical experiments, we evaluate these structure on simulated simple pendulum and CartPole systems and show that incorporating these characteristics into the model structure leads to accurate learning with a smaller amount of training data compared to the baseline structure.
In this paper, we present controllers to stabilize benchmark under-actuated systems such as inertia wheel pendulum (IWP), cart-pole, cart-pendulum, overhead crane, VTOL aircraft, ball-and-beam, ...translational oscillations with a rotational actuator (TORA), and active magnetic bearing (AMB). The mentioned systems are transformed into the feedforward systems subject to uncertain gains, the nested-saturation controllers in an almost unified form are then designed. Due to the analysis mechanism of saturation reduction (checking the time-derivatives of boundary surfaces), the bounds of uncertain gains can always be determined in small domains. In turn, there is no need to treat in advance certain subsystems, and the fully-saturated controllers (rather than partially-saturated ones) can be taken as global stabilizers. Moreover, the explicit inequality conditions can always be summarized. In addition, for the relatively complicated systems such as the overhead crane system, a non-strict upper-triangular subsystem of the ball-and-beam system as well as the TORA system, special technical strategies are developed to treat the linear perturbation, the non-strict upper-triangular structure and an oscillator.
This research reports on the recent development of black-box optimization methods based on single-step deep reinforcement learning and their conceptual similarity to evolution strategy (ES) ...techniques. It formally introduces policy-based optimization (PBO), a policy-gradient-based optimization algorithm that relies on a policy network to describe the density function of its forthcoming evaluations, and uses covariance estimation to steer the policy improvement process in the right direction. The specifics of the PBO algorithm are detailed, and the connections to evolutionary strategies are discussed. Relevance is assessed by benchmarking PBO against classical ES techniques on analytic functions minimization problems, and by optimizing various parametric control laws intended for the Lorenz attractor and the classical cartpole problem. Given the scarce existing literature on the topic, this contribution definitely establishes PBO as a valid, versatile black-box optimization technique, and opens the way to multiple future improvements building on the inherent flexibility of the neural networks approach.
In this paper, to solve nonlinear optimal control problems which can be modeled as Markov decision processes (MDPs), we present an online self-learning control algorithm called dual heuristic ...programming with global Laplacian Eigenmaps (GLEM-DHP). The GLEM-DHP algorithm makes use of GLEM, which is an improved manifold learning approach with global information, to learn the features for value function approximation of the MDP. Different from traditional feature representation methods using neural networks, the manifold-based features can be learned before the online learning process by collecting samples from the MDP. More importantly, in addition to local features, global information can also be utilized by using the geodesic minimum spanning tree (GMST) approach. Based on the theoretical property of GMST, it is shown that the GLEM-based features can represent the intrinsic geometric property of MDP states, which is beneficial to improve the performance of value function approximation and, hence, leads to better learning control properties. To compare the proposed method with previous learning control algorithms, the performance of GLEM-DHP is evaluated on two nonlinear control problems, which include the cart-pole problem and the ball-plate control problem. Simulation and experimental results show that the GLEM-DHP algorithm can obtain better learning control performance than previous learning control algorithms with manually designed features, as well as manifold features only with local information.
PDF
