On average, men and women differ in brain structure and behavior, raising the possibility of a link between sex differences in brain and behavior. But women and men are also subject to different societal and cultural norms. We navigated this challenge by investigating variability of sex-differentiated brain structure within each sex. Using data from the Queensland Twin IMaging study (n = 1,040) and Human Connectome Project (n = 1,113), we obtained data-driven measures of individual differences along a male–female dimension for brain and behavior based on average sex differences in brain structure and behavior, respectively. We found a weak association between these brain and behavioral differences, driven by brain size. These brain and behavioral differences were moderately heritable. Our findings suggest that behavioral sex differences are, to some extent, related to sex differences in brain structure but that this is mainly driven by differences in brain size, and causality should be interpreted cautiously.