This paper introduces a novel algorithm for optimizing the coefficients of the digital filters used in incremental delta-sigma analog-to-digital converters (IDSC). This algorithm is modified from constrained linear least squares (LS) to improve the signal-to-noise-and-distortion ratio (SNDR) of IDSC and minimize the oversampling rate (OSR) of the modulator, which enhances system speed and reduces power consumption. In the case of the same SNDR, the first-order IDSC with the proposed filter can reduce the OSR of the modulator by 50%, at least compared to the IDSC with the conventional filter. The second-order IDSC with the proposed filter can reach a higher SNDR of 106 dB with an OSR of 64. Considering the nonlinearity of the integrator, the SNDR of IDSC with the proposed filter is also 10 dB greater than that of the conventional filters and 3dB greater than that of the IDSC using the near-optimal algorithm filter. The experimental results indicate that the proposed filter possesses an excellent figure of merit of 0.028  pJ / conv .