Only five species of the once-diverse Rhinocerotidae remain, making the reconstruction of their evolutionary history a challenge to biologists since Darwin. We sequenced genomes from five rhinoceros species (three extinct and two living), which we compared to existing data from the remaining three living species and a range of outgroups. We identify an early divergence between extant African and Eurasian lineages, resolving a key debate regarding the phylogeny of extant rhinoceroses. This early Miocene (∼16 million years ago mya) split post-dates the land bridge formation between the Afro-Arabian and Eurasian landmasses. Our analyses also show that while rhinoceros genomes in general exhibit low levels of genome-wide diversity, heterozygosity is lowest and inbreeding is highest in the modern species. These results suggest that while low genetic diversity is a long-term feature of the family, it has been particularly exacerbated recently, likely reflecting recent anthropogenic-driven population declines. Display omitted •Analysis of genomes from all five extant and three extinct rhinoceros species•Strong phylogenomic support for the geographical hypothesis of rhinoceros evolution•Basal split between African and Eurasian lineages in the early Miocene (∼16 mya)•While all rhinoceroses have low genome diversity, it is lowest in modern-day ones The comparison of de novo genomes from the white, black, Sumatran, and greater one-horned rhinoceroses with the genomes of a historic Javan rhinoceros and three extinct Pleistocene species resolves the evolutionary relationships within the Rhinocerotidae family and reveals that low genetic diversity is a long-term feature of rhinoceroses.