In magic-angle twisted bilayer graphene, the moiré superlattice potential gives rise to narrow electronic bands that support a multitude of many-body quantum phases. Further richness arises in the presence of a perpendicular magnetic field, where the interplay between moiré and magnetic length scales leads to fractal Hofstadter subbands. In this strongly correlated Hofstadter platform, multiple experiments have identified gapped topological and correlated states, but little is known about the phase transitions between them in the intervening compressible regimes. Here we simultaneously unveil sequences of broken-symmetry Chern insulators and resolve sharp phase transitions between competing states with different topological quantum numbers and different occupations of the spin-valley flavour. Our measurements determine the energy spectrum of interacting Hofstadter subbands in magic-angle twisted bilayer graphene and map out the phase diagram of flavour occupancy. In addition, we observe full lifting of the degeneracy of the zeroth Landau levels together with level crossings, indicating moiré valley splitting. We propose a unified flavour polarization mechanism to understand the intricate interplay of topology, interactions and symmetry breaking as a function of density and applied magnetic field in this system.In graphene, the spin and valley degrees of freedom combine into a higher-order isospin. Now, a full map of the phase diagram of this isospin is measured in the moiré bands of twisted bilayer graphene.