Social memory, the ability to recognize and remember individuals within a social group, is crucial for social interactions and relationships. Deficits in social memory have been linked to several neuropsychiatric and neurodegenerative disorders. The hippocampus, especially the circuit that links dorsal CA2 and ventral CA1 neurons, is considered a neural substrate for social memory formation. Recent studies have provided compelling evidence of extrahippocampal contributions to social memory. The septal nuclei, including the medial and lateral septum, make up a basal forebrain region that shares bidirectional neuronal connections with the hippocampus and has recently been identified as critical for social memory. The focus of our review is the neural circuit mechanisms that underlie social memory, with a special emphasis on the septum. We also discuss the social memory dysfunction associated with neuropsychiatric and neurodegenerative disorders.