Conspicuous coloration displayed by animals that express sexual colour dimorphism is generally explained as an adaptation to sexual selection, yet the interactions and relative effects of selective forces influencing colour dimorphism are largely unknown. Qualitatively, colour dimorphism appears more pronounced in marine fishes that live on coral reefs where traits associated with strong sexual selection are purportedly more common. Using phylogenetic comparative analysis, we show that wrasses and parrotfishes exclusive to coral reefs are the most colour dimorphic, but surprisingly, the effect of habitat is not influenced by traits associated with strong sexual selection. Rather, habitat-specific selective forces, including clear water and structural refuge, promote the evolution of pronounced colour dimorphism that manifests colours less likely to be displayed in other habitats. Our results demonstrate that environmental context ultimately determines the evolution of conspicuous coloration in colour-dimorphic labrid fishes, despite other influential selective forces.