Abstract We use deep Chandra X-ray imaging to measure the distribution of specific black hole accretion rates (LX relative to the stellar mass of the galaxy) and thus trace active galactic nucleus (AGN) activity within star-forming and quiescent galaxies, as a function of stellar mass (from 108.5 to 1011.5 M⊙) and redshift (to z ∼ 4). We adopt near-infrared-selected samples of galaxies from the CANDELS and UltraVISTA surveys, extract X-ray data for every galaxy, and use a flexible Bayesian method to combine these data and to measure the probability distribution function of specific black hole accretion rates, λsBHAR. We identify a broad distribution of λsBHAR in both star-forming and quiescent galaxies – likely reflecting the stochastic nature of AGN fuelling – with a roughly power-law shape that rises towards lower λsBHAR, a steep cut-off at λsBHAR ≳ 0.1–1 (in Eddington equivalent units), and a turnover or flattening at $\lambda _\mathrm{sBHAR} \lesssim 10^{-3}\hbox{ {to} }10^{-2}$. We find that the probability of a star-forming galaxy hosting a moderate λsBHAR AGN depends on stellar mass and evolves with redshift, shifting towards higher λsBHAR at higher redshifts. This evolution is truncated at a point corresponding to the Eddington limit, indicating black holes may self-regulate their growth at high redshifts when copious gas is available. The probability of a quiescent galaxy hosting an AGN is generally lower than that of a star-forming galaxy, shows signs of suppression at the highest stellar masses and evolves strongly with redshift. The AGN duty cycle in high-redshift (z ≳ 2) quiescent galaxies thus reaches ∼20 per cent, comparable to the duty cycle in star-forming galaxies of equivalent stellar mass and redshift.