On Nonconvex Decentralized Gradient Descent Zeng, Jinshan; Yin, Wotao
IEEE transactions on signal processing,
2018-June1,-1, 2018-6-1, Volume:
66, Issue:
11
Journal Article
Peer reviewed
Open access
Consensus optimization has received considerable attention in recent years. A number of decentralized algorithms have been proposed for convex consensus optimization. However, to the behaviors or ...consensus nonconvex optimization, our understanding is more limited. When we lose convexity, we cannot hope that our algorithms always return global solutions though they sometimes still do. Somewhat surprisingly, the decentralized consensus algorithms, DGD and Prox-DGD, retain most other properties that are known in the convex setting. In particular, when diminishing (or constant) step sizes are used, we can prove convergence to a (or a neighborhood of) consensus stationary solution under some regular assumptions. It is worth noting that Prox-DGD can handle nonconvex nonsmooth functions if their proximal operators can be computed. Such functions include SCAD, MCP, and ℓ quasi-norms, q ∈ 0,1). Similarly, Prox-DGD can take the constraint to a nonconvex set with an easy projection. To establish these properties, we have to introduce a completely different line of analysis, as well as modify existing proofs that were used in the convex setting.
•A novel cavity with two rotary waveguides for efficient heating is developed.•The gradient descent method is used to improve the uniformity of heating object.•An experimental system is established ...and the simulation results are verified.•Under this method, the shape, position, and material of the object are discussed.
This study aims to improve the efficiency and uniformity of microwave heating based on a rotary radiation structure. This study’s steps are summarized as follows. First, using the finite element method in a simulation model, two radiation ports revolve on their axes and heat the material. Second, the heating efficiency at different angles is analyzed to find efficient angles of radiation ports. Finally, to improve the uniformity of material with high efficiency, we use the gradient descent method in machine learning to optimize the heating time at selected angles. An experimental system is established for this research and the simulation results are verified by experiments, showing that this research has an obvious superiority and efficiency and uniformity of the material have improved compared to traditional heating results without angle selection and optimization of the heating time. What is more, influences of material with different shapes, positions and materials have also been discussed.
For the existing repetitive motion generation (RMG) schemes for kinematic control of redundant manipulators, the position error always exists and fluctuates. This article gives an answer to this ...phenomenon and presents the theoretical analyses to reveal that the existing RMG schemes exist a theoretical position error related to the joint angle error. To remedy this weakness of existing solutions, an orthogonal projection RMG (OPRMG) scheme is proposed in this article by introducing an orthogonal projection method with the position error eliminated theoretically, which decouples the joint space error and Cartesian space error with joint constraints considered. The corresponding new recurrent neural networks (NRNNs) are structured by exploiting the gradient descent method with the assistance of velocity compensation with theoretical analyses provided to embody the stability and feasibility. In addition, simulation results on a fixed-based redundant manipulator, a mobile manipulator, and a multirobot system synthesized by the existing RMG schemes and the proposed one are presented to verify the superiority and precise performance of the OPRMG scheme for kinematic control of redundant manipulators. Moreover, via adjusting the coefficient, simulations on the position error and joint drift of the redundant manipulator are conducted for comparison to prove the high performance of the OPRMG scheme. To bring out the crucial point, different controllers for the redundancy resolution of redundant manipulators are compared to highlight the superiority and advantage of the proposed NRNN. This work greatly improves the existing RMG solutions in theoretically eliminating the position error and joint drift, which is of significant contributions to increasing the accuracy and efficiency of high-precision instruments in manufacturing production.
In this paper, based on the Karush-Kuhn-Tucker (KKT) conditions of
regularization, we propose a communication-efficient distributed learning approach for high-dimensional and sparse generalized ...linear models with massive data sets stored across different machines. This proposed method is a support detection and root finding method for generalized linear models in a distributed form. In each round of the proposed method, the support set is first determined by the primal and dual information reduced to the master machine, then the reduced maximum likelihood estimator is obtained by the gradient descent method, among which it only suffices to calculate the gradient vectors on each machine and communicate them instead of the data. We give the optimal
-norm error bound for the sequences generated by the proposed algorithm and show that this
-norm error bound decays exponentially to the optimal order. Moreover, we show that the oracle estimator can be recovered if the target signal is not less than the detectable level. In addition, an adaptive version of the proposed algorithm is developed to estimate the sparsity level. Simulation studies illustrate the superior performance of the proposed methods.
Efficiently enhancing heat conduction through optimized distribution of a limited quantity of high thermal conductivity material is paramount in cooling electronic devices and numerous other ...applications. This paper introduces a target-driven all-at-once approach for PDE-constrained optimization and derives a splitting smoothed particle hydrodynamics (SPH) method for optimizing the distribution of thermal conductivity in heat conduction problems. In this method, the optimization iteration of the system is split into several easily addressed steps. A targeting step is employed to progressively enforce the direct target, which potentially leads to increased PDE residuals. Then, these residuals are recovered through an evolution step of the design variable. After this, a PDE solution step is carried out to further decrease the PDE residuals, and the system is ready for the next iteration. Unlike the simulation-based approaches, the present method does not rely on the adjoint state equation and converged state variable field in each iteration, and the optimization process is significantly simplified and accelerated. With the utilization of an implicit SPH splitting operator and a general numerical regularization formulation, the information propagation is further accelerated and the numerical stability is greatly enhanced. Typical examples of heat conduction optimization demonstrate that the current method yields optimal results comparable to previous methods and exhibits considerable computational efficiency. Moreover, the optimal results feature more moderate extreme values, which offers distinct advantages for the easier selection of appropriate material with high thermal conductivity.
•The optimization of the heat conduction is split into several easily addressed weakly coupled steps.•The target is directly imposed on the temperature, and fields only converge at the termination.•The splitting SPH method is employed to implicitly update temperature and thermal conductivity.•General regularization formulation is proposed to guarantee numerical stability.•Optimal results with moderate peak values of thermal conductivity are obtained with good efficiency.
On the basis of the barrier method, a unified approach for the static output feedback sliding mode control of linear control systems with matched uncertainty is addressed. Without coordinate ...transformations, a structured static output feedback sliding surface matrix is obtained by solving a constrained minimization problem, and then a globally stabilizing static output feedback sliding mode controller can be constructed directly. The Lagrange multiplier method is used to derive the necessary conditions for the optimal solution of the minimization problem. An iterative gradient descent algorithm is established to search for a (local) optimal solution. Finally, for validation, a numerical example is proposed.
The goal of tensor completion is to fill in missing entries of a partially known tensor under a low-rank constraint. In this paper, we study low rank third-order tensor completion problems by using ...Riemannian optimization methods on the smooth manifold. Here the tensor rank is defined to be a multi-rank under the Discrete Cosine Transform-related transform tensor-tensor product. With suitable incoherence conditions on the underlying low multi-rank rank tensor, we show that the proposed Riemannian optimization method is guaranteed to converge to the underlying low multi-rank tensor with a high probability. Number of sampling entries required for convergence are also derived. Numerical examples of synthetic data and real data sets are reported to demonstrate that the performance of the proposed method is better than that of tensor-based method using the Tucker-rank model in terms of computational time, and that of the matrix-based completion method in terms of number of sampling entries.
Making optical networks more efficient and reliable requires further automation of the optical layer. In this context, we propose a closed control loop that automatically performs fine frequency ...adjustments of the subchannels of superchannels to maintain optimal performance despite time-dependent impairments, thus achieving three main goals. At design, our scheme reduces the need for margins and guard-bands dedicated to spectrum-related impairments such as filter or channel detunings. This allows for a more efficient network at deployment. Then, during operation, the quality of transmission (QoT) of each subchannel is maximized to make the superchannel more resilient to any kind of soft failure, thus improving network resilience. Finally, considering the elastic network paradigm, performance improvements can be converted into significant capacity upgrades. We demonstrate the ability of our solution to achieve these three goals with split-step Fourier simulations. For a four-subchannel superchannel, we demonstrate robustness to +/- 2GHz frequency detunings and gains of up to 3.7 dB in quality of transmission.
This article presents a novel fractional order LMS (FOLMS) algorithm, which involves a variable gradient order scheme. The fractional order gradient descent method is revisited firstly. A variable ...initial value scheme is proposed to attenuate the non–locality of fractional order calculus and to ensure the convergence of the proposed FOLMS algorithm. Furthermore, it is noticed that a contradiction between rapidity and accuracy always appears together with the advancement of FOLMS algorithm; namely, a larger value of the gradient order can not only give a faster convergence speed, but also correspond to a larger estimation error. For the purpose of removing the contradiction between rapidity and accuracy, a variable gradient order scheme is designed for the FOLMS algorithm. Based on a sufficient large number of independent runs, the efficiency and superiority of the proposed algorithm are demonstrated in numerical examples finally.
•Fractional order gradient decent method with variable initial value is developed.•Fractional order LMS (FOLMS) with variable initial value is proposed.•The performance of the proposed FOLMS is analyzed.•FOLMS with variable gradient order is constructed.
•A novel fractional order gradient descent method based on quadratic loss function is proposed.•A novel two-stage fractional order gradient descent method is proposed with random weight particle ...swarm optimization algorithm.•The scope of application of the fractional order gradient descent is expanded.
This article introduces a novel fractional order gradient descent method for the quadratic loss function. Based on Riemann-Liouville definition, a more practical fractional order gradient descent method with variable initial value is proposed to ensure convergence to the actual extremum. On this basis, the random weight particle swarm optimization algorithm is introduced to select the appropriate initial value, which not only accelerates the convergence speed, but also enhances the global convergence ability of the algorithm. To avoid complicated problems of the chain rule in fractional calculus, the parameters of output layers is trained by the new designed method, while the parameters of hidden layers still use the conventional method. By selecting proper hyper-parameters, the proposed method shows faster convergence speed than others. Finally, numerical examples are given to verify that the proposed algorithm has fast convergence speed and high accuracy under a adequate large number of independent runs.
PDF
