Objective To investigate how the presence/side of hippocampal sclerosis (HS) are related to the white matter structure of cingulum bundle (CB), arcuate fasciculus (AF), and inferior longitudinal fasciculus (ILF) in mesial temporal lobe epilepsy (MTLE). Methods We acquired diffusion‐weighted magnetic resonance imaging (MRI) from 86 healthy and 71 individuals with MTLE (22 righ‐HS; right‐HS, 34 left‐HS; left‐HS, and 15 nonlesional MTLE). We utilized two‐tensor tractography and fiber clustering to compare fractional anisotropy (FA) of each side/tract between groups. Additionally, we examined the association between FA and nonverbal (WMS‐R) and verbal (WMS‐R, RAVLT codification) memory performance for MTLE individuals. Results White matter abnormalities depended on the side and presence of HS. The left‐HS demonstrated widespread abnormalities for all tracts, the right‐HS showed lower FA for ipsilateral tracts and the nonlesional MTLE group did not differ from healthy individuals. Results indicate no differences in verbal/nonverbal memory performance between the groups, but trend‐level associations between higher FA of visual memory and the left CB (r = 0.286, P = 0.018), verbal memory (RAVLT) and ‐left CB (r = 0.335, P = 0.005), ‐right CB (r = 0.286, P = 0.016), and ‐left AF (r = 0.287, P = 0.017). Significance Our results highlight that the presence and side of HS are crucial to understand the pathophysiology of MTLE. Specifically, left‐sided HS seems to be related to widespread bilateral white matter abnormalities. Future longitudinal studies should focus on developing diagnostic and treatment strategies dependent on HS's presence/side.