The temporal association cortex is considered a primate specialization and is involved in complex behaviors, with some, such as language, particularly characteristic of humans. The emergence of these behaviors has been linked to major differences in temporal lobe white matter in humans compared with monkeys. It is unknown, however, how the organization of the temporal lobe differs across several anthropoid primates. Therefore, we systematically compared the organization of the major temporal lobe white matter tracts in the human, gorilla, and chimpanzee great apes and in the macaque monkey. We show that humans and great apes, in particular the chimpanzee, exhibit an expanded and more complex occipital-temporal white matter system; additionally, in humans, the invasion of dorsal tracts into the temporal lobe provides a further specialization. We demonstrate the reorganization of different tracts along the primate evolutionary tree, including distinctive connectivity of human temporal gray matter.