Color is continuous, yet we group colors into discrete categories associated with color names (e.g., yellow, blue). Color categorization is a case in point in the debate on how language shapes human cognition. Evidence suggests that color categorization depends on top-down input from the language system to the visual cortex. We directly tested this hypothesis by assessing color categorization in a stroke patient, RDS, with a rare, selective deficit in naming visually presented chromatic colors, and relatively preserved achromatic color naming. Multimodal MRI revealed a left occipito-temporal lesion that directly damaged left color-biased regions, and functionally disconnected their right-hemisphere homologs from the language system. The lesion had a greater effect on RDS’s chromatic color naming than on color categorization, which was relatively preserved on a nonverbal task. Color categorization and naming can thus be independent in the human brain, challenging the mandatory involvement of language in adult human cognition. Display omitted •Are color categories an example of linguistic impact on non-verbal human cognition?•Patient RDS shows impairment in naming colors after a left-hemisphere stroke•The lesion disconnects color-biased visual regions from the language system•Relative sparing of color categorization shows its independence from color naming Color categories (e.g., red, yellow) may result from the top-down impact of language on perception. Siuda-Krzywicka et al. describe a patient with impaired color naming, after a stroke disconnects color perception from language. The patient still categorizes colors they could not name, showing robustness of color categorization against impaired linguistic processing.