We present the results of a search for elastic scattering from galactic dark matter in the form of Weakly Interacting Massive Particles (WIMPs) in the 4-30 GeV/$c^2$ mass range. We make use of a 582 kg-day fiducial exposure from an array of 800 g Germanium bolometers equipped with a set of interleaved electrodes with full surface coverage. We searched specifically for $\sim 2.5-20$ keV nuclear recoils inside the detector fiducial volume. As an illustration the number of observed events in the search for 5 (resp. 20) GeV/$c^2$ WIMPs are 9 (resp. 4), compared to an expected background of 6.1 (resp. 1.4). A 90% CL limit of $4.3\times 10^{-40}$ cm$^2$ (resp. $9.4\times 10^{-44}$ cm$^2$) is set on the spin-independent WIMP-nucleon scattering cross-section for 5 (resp. 20) GeV/$c^2$ WIMPs. This result represents a 41-fold improvement with respect to the previous EDELWEISS-II low-mass WIMP search for 7 GeV/$c^2$ WIMPs. The derived constraint is in tension with hints of WIMP signals from some recent experiments, thus confirming results obtained with different detection techniques.