The impact of ASD on parental QOL was evaluated in the EpiTED cohort study at early adulthood. Two-third of parents of young adults with ASD (66.7%) reported that their QoL was at least moderately altered. The perceived impact of ASD on parental QoL was related to the young adults’ level of adaptive skills, as well as to symptom severity and the presence of challenging behaviors, which appeared to be the main risk factor. The study of change between adolescence and early adulthood showed that parents whose children had a decrease in challenging behaviors perceived a decreased impact on their QoL. These results argue for the importance to propose specific interventions to target associated challenging behaviors in ASD.