Careful study of the clinical outcomes of temporal lobe epilepsy (TLE) surgery has greatly advanced our knowledge of the neuroanatomy of human memory. After early cases resulted in profound amnesia, the critical role of the hippocampus and associated medial temporal lobe (MTL) structures to declarative memory became evident. Surgical approaches quickly changed to become unilateral and later, to be more precise, potentially reducing cognitive morbidity. Neuropsychological studies following unilateral temporal lobe resection (TLR) have challenged early models, which simplified the lateralization of verbal and visual memory function. Diagnostic tests, including intracarotid sodium amobarbital procedure (WADA), structural magnetic resonance imaging (MRI), and functional neuroimaging (functional MRI (fMRI), positron emission tomography (PET), and single-photon emission computed tomography (SPECT)), can more accurately lateralize and localize epileptogenic cortex and predict memory outcomes from surgery. Longitudinal studies have shown that memory may even improve in seizure-free patients. From 70 years of experience with epilepsy surgery, we now have a richer understanding of the clinical, neuroimaging, and surgical predictors of memory decline—and improvement—after TLR. "Special Issue: Epilepsy & Behavior's 20th Anniversary" •Early cases of amnesia after temporal lobe surgery revealed the critical function of mesial temporal lobe to memory.•Tracking post-operative cognition after epilepsy surgery has yielded further insight into the neuroanatomy of memory.•Newer surgical approaches and improved diagnostic testing can reduce risk of post-operative memory decline.•Some cases of improved post-operative memory reveal the harmful effect of ongoing seizures and potential for recovery.