Models of person perception have long asserted that our impressions of others are guided by characteristics of both the target and perceiver. However, research has not yet quantified to what extent perceivers and targets contribute to different impressions. This quantification is theoretically critical, as it addresses how much an impression arises from "our minds" versus "others' faces." Here, we apply cross-classified random effects models to address this fundamental question in social cognition, using approximately 700,000 ratings of faces. With this approach, we demonstrate that (a) different trait impressions have unique causal processes, meaning that some impressions are largely informed by perceiver-level characteristics whereas others are driven more by physical target-level characteristics; (b) modeling of perceiver- and target-variance in impressions informs fundamental models of social perception; (c) Perceiver × Target interactions explain a substantial portion of variance in impressions; (d) greater emotional intensity in stimuli decreases the influence of the perceiver; and (e) more variable, naturalistic stimuli increases variation across perceivers. Important overarching patterns emerged. Broadly, traits and dimensions representing inferences of character (e.g., dominance) are driven more by perceiver characteristics than those representing appearance-based appraisals (e.g., youthful-attractiveness). Moreover, inferences made of more ambiguous traits (e.g., creative) or displays (e.g., faces with less extreme emotions, less-controlled stimuli) are similarly driven more by perceiver than target characteristics. Together, results highlight the large role that perceiver and target variability play in trait impressions, and develop a new topography of trait impressions that considers the source of the impression.