Prosopagnosia is a cognitive disorder in which facial recognition is severely impaired despite normal vision and intelligence. Prosopagnosia was first reported in the 1800s, but its cause remains unclear. Although other neurological symptoms are often present, some patients have pure prosopagnosia. The bilateral occipital lobes are believed to be associated with symptoms. Recent brain imaging techniques have identified the right fusiform gyrus (rFG), located at the junction of the right occipital temporal lobe, as the affected region. In this report, we present a case of associative prosopagnosia with no concomitant symptoms in a 76-year-old man. Brain magnetic resonance imaging detected a subcortical hemorrhage in the right temporal lobe. Using tractography based on diffusion tensor imaging, we visualized atrophy of the right inferior longitudinal fasciculus (ILF). This is the first time tractography has been used to show a clear association between associative prosopagnosia and ILF damage projecting from the rFG.