ABSTRACT We compare the results of a large grid of N-body simulations with the surface brightness and velocity dispersion profiles of the globular clusters ω Cen and NGC 6624. Our models include clusters with varying stellar-mass black hole retention fractions and varying masses of a central intermediate-mass black hole (IMBH). We find that an $\sim 45\, 000$ M⊙ IMBH, whose presence has been suggested based on the measured velocity dispersion profile of ω Cen, predicts the existence of about 20 fast-moving, m > 0.5 M⊙, main-sequence stars with a (1D) velocity v > 60 km s−1 in the central 20 arcsec of ω Cen. However, no such star is present in the HST/ACS proper motion catalogue of Bellini et al. (2017), strongly ruling out the presence of a massive IMBH in the core of ω Cen. Instead, we find that all available data can be fitted by a model that contains 4.6 per cent of the mass of ω Cen in a centrally concentrated cluster of stellar-mass black holes. We show that this mass fraction in stellar-mass BHs is compatible with the predictions of stellar evolution models of massive stars. We also compare our grid of N-body simulations with NGC 6624, a cluster recently claimed to harbour a 20 000 M⊙ black hole based on timing observations of millisecond pulsars. However, we find that models with MIMBH > 1000 M⊙ IMBHs are incompatible with the observed velocity dispersion and surface brightness profile of NGC 6624, ruling out the presence of a massive IMBH in this cluster. Models without an IMBH provide again an excellent fit to NGC 6624.