The embodied cognition paradigm has stimulated ongoing debate about whether sensory data – including color – contributes to the semantic structure of abstract concepts. Recent uses of linguistic data in the study of embodied cognition have been focused on textual corpora, which largely precludes the direct analysis of sensory information. Here, we develop an automated approach to multimodal content analysis that detects associations between words based on the color distributions of their Google Image search results. Crucially, we measure color using a transformation of colorspace that closely resembles human color perception. We find that words in the abstract domains of academic disciplines, emotions, and music genres, cluster in a statistically significant fashion according to their color distributions. Furthermore, we use the lexical ontology WordNet and crowdsourced human judgments to show that this clustering reflects non-arbitrary semantic structure, consistent with metaphor-based accounts of embodied cognition. In particular, we find that images corresponding to more abstract words exhibit higher variability in colorspace, and semantically similar words have more similar color distributions. Strikingly, we show that color associations often reflect shared affective dimensions between abstract domains, thus revealing patterns of aesthetic coherence in everyday language. We argue that these findings provide a novel way to synthesize metaphor-based and affect-based accounts of embodied semantics.