Eukaryotic plasma membranes are compartmentalized into functional lateral domains, including lipid-driven membrane rafts. Rafts are involved in most plasma membrane functions by selective recruitment and retention of specific proteins. However, the structural determinants of transmembrane protein partitioning to raft domains are not fully understood. Hypothesizing that protein transmembrane domains (TMDs) determine raft association, here we directly quantify raft affinity for dozens of TMDs. We identify three physical features that independently affect raft partitioning, namely TMD surface area, length, and palmitoylation. We rationalize these findings into a mechanistic, physical model that predicts raft affinity from the protein sequence. Application of these concepts to the human proteome reveals that plasma membrane proteins have higher raft affinity than those of intracellular membranes, consistent with raft-mediated plasma membrane sorting. Overall, our experimental observations and physical model establish general rules for raft partitioning of TMDs and support the central role of rafts in membrane traffic.