Context. Close encounters with (1) Ceres and (4) Vesta, the two most massive bodies in the main belt, are known to be a mechanism of dynamical mobility able to significantly alter proper elements of minor bodies, and they are the main source of dynamical mobility for medium-sized and large asteroids (D > 20 km, approximately). Recently, it has been shown that drift rates caused by close encounters with massive asteroids may change significantly on timescales of 30 Myr when different models (i.e., different numbers of massive asteroids) are considered. Aims. So far, not much attention has been given to the case of diffusion caused by the other most massive bodies in the main belt: (2) Pallas, (10) Hygiea, and (31) Euphrosyne, the third, fourth, and one of the most massive highly inclined asteroids in the main belt, respectively. Since (2) Pallas is a highly inclined object, relative velocities at encounter with other asteroids tend to be high and changes in proper elements are therefore relatively small. It was thus believed that the scattering effect caused by highly inclined objects in general should be small. Can diffusion by close encounters with these asteroids be a significant mechanism of long-term dynamical mobility? Methods. By performing simulations with symplectic integrators, we studied the problem of scattering caused by close encounters with (2) Pallas, (10) Hygiea, and (31) Euphrosyne when only the massive asteroids (and the eight planets) are considered, and the other massive main belt asteroids and non-gravitational forces are also accounted for. Results. By finding relatively small values of drift rates for (2) Pallas, we confirm that orbital scattering by this highly inclined object is indeed a minor effect. Unexpectedly, however, we obtained values of drift rates for changes in proper semi-major axis a caused by (10) Hygiea and (31) Euphrosyne larger than what was previously found for scattering by (4) Vesta. These high rates may have repercussions on the orbital evolution and age estimate of their respective families.