Summary It is significant to fit a Gaussian function with the observation data for artificial intelligence or other engineering fields. Considering the influence of noises, this article proposes a nonlinear optimization method for fitting the Gaussian activation functions. By means of the gradient search and the Newton search, a direct gradient‐based iterative algorithm and a direct Newton iterative algorithm are presented for identifying the Gaussian functions. Considering the computational cost, the authors develop a multi‐innovation stochastic gradient algorithm for the noisy Gaussian functions. After introducing a forgetting factor, the parameter estimation accuracy can be further improved. The simulation results indicate that the proposed nonlinear optimization method and gradient‐based algorithms can fit the noisy Gaussian functions very well.