In this paper, we show how tomographic imaging (Zeeman–Doppler imaging, ZDI) can be used to characterize stellar activity and magnetic field topologies, ultimately allowing us to filter out the radial velocity (RV) activity jitter of M dwarf moderate rotators. This work is based on spectropolarimetric observations of a sample of five weakly active early-M dwarfs (GJ 205, GJ 358, GJ 410, GJ 479, GJ 846) with HARPS-Pol and NARVAL. These stars have v sin i and RV jitters in the range 1–2 km s−1 and 2.7–10.0 m s−1 rms, respectively. Using a modified version of ZDI applied to sets of phase-resolved least-squares deconvolved profiles of unpolarized spectral lines, we are able to characterize the distribution of active regions at the stellar surfaces. We find that dark spots cover less than 2 per cent of the total surface of the stars of our sample. Our technique is efficient at modelling the rotationally modulated component of the activity jitter, and succeeds at decreasing the amplitude of this component by typical factors of 2–3 and up to 6 in optimal cases. From the rotationally modulated time series of circularly polarized spectra and with ZDI, we also reconstruct the large-scale magnetic field topology. These fields suggest that bistability of dynamo processes observed in active M dwarfs may also be at work for moderately active M dwarfs. Comparing spot distributions with field topologies suggest that dark spots causing activity jitter concentrate at the magnetic pole and/or equator, to be confirmed with future data on a larger sample.