Abstract The dynamics of black hole (BH) seeds in high-redshift galaxies is key to understand their ability to grow via accretion and to pair in close binaries during galactic mergers. To properly follow the dynamics of BHs we develop a physically motivated model to capture unresolved dynamical friction from stars, dark matter, and gas. We first validate the model and then we use it to investigate the dynamics of seed BHs born at z ∼ 9 in dwarf proto-galaxies. We perform a suite of zoom cosmological simulations with spatial resolution as high as 10 pc and with a stellar and dark matter mass resolution of $2\times 10^3 \, \, $ and $2\times 10^5 \, \, \mathrm{ M}_{\odot }$, respectively. We first explore the dynamics of a seed BH in the galaxy where it is born and show that it is highly erratic if the seed mass is less than $10^5\, \, \mathrm{ M}_{\odot }$. The dynamics is dominated by the stellar component, whose distribution is irregular and patchy, thus inducing stochasticity in the orbits: the BH may be anywhere in the proto-galaxy. When this dwarf merges into a larger galaxy, it is paramount to simulate the process with very high spatial and mass resolution in order to correctly account for the stripping of the stellar envelope of the satellite BH. The outcome of the encounter could be either a tight binary or, at least temporary, a wandering BH, leading to multiple BHs in a galaxy, each inherited from a different merger.