Background: Cholinergic deficiency has been suggested to associate with the abnormal accumulation of Aβ and tau for patients with Alzheimer's disease (AD). However, no studies have investigated the effect of APOE -ε4 and group differences in modulating the cholinergic basal forebrain–amygdala network for subjects with different levels of cognitive impairment. We evaluated the effect of APOE -ε4 on the cholinergic structural association and the neurocognitive performance for subjects with different levels of cognitive impairment. Methods: We used the structural brain magnetic resonance imaging scans from the Alzheimer's Disease Neuroimaging Initiative dataset. The study included cognitively normal (CN, n = 167) subjects and subjects with significant memory concern (SMC, n = 96), early mild cognitive impairment (EMCI, n = 146), late cognitive impairment (LMCI, n = 138), and AD ( n = 121). Subjects were further categorized according to the APOE -ε4 allele carrier status. The main effects of APOE -ε4 and group difference on the brain volumetric measurements were assessed. Regression analyses were conducted to evaluate the associations among cholinergic structural changes, APOE -ε4 status, and cognitive performance. Results: We found that APOE -ε4 carriers in the disease group showed higher brain atrophy than non-carriers in the cholinergic pathway, while there is no difference between carriers and non-carriers in the CN group. APOE -ε4 allele carriers in the disease groups also exhibited a stronger cholinergic structural correlation than non-carriers did, while there is no difference between the carriers and non-carriers in the CN subjects. Disease subjects exhibited a stronger structural correlation in the cholinergic pathway than CN subjects did. Moreover, APOE- ε4 allele carriers in the disease group exhibited a stronger correlation between the volumetric changes and cognitive performance than non-carriers did, while there is no difference between carriers and non-carriers in CN subjects. Disease subjects exhibited a stronger correlation between the volumetric changes and cognitive performance than CN subjects did. Conclusion: Our results confirmed the effect of APOE- ε4 on and group differences in the associations with the cholinergic structural changes that may reflect impaired brain function underlying neurocognitive degeneration in AD.