Younger siblings of children with autism spectrum disorder (ASD) are themselves at increased risk for ASD and other developmental concerns. It is unclear if infants who display developmental concerns, but are unaffected by ASD, share similar or dissimilar behavioral and brain phenotypes to infants with ASD. Most individuals with ASD exhibit heterogeneous difficulties with language, and their receptive-expressive language profiles are often atypical. Yet, little is known about the neurobiology that contributes to these language difficulties. In this study, we used behavioral assessments and structural magnetic resonance imaging to investigate early brain structures and associations with later language skills. High-risk infants who were later diagnosed with ASD (n = 86) were compared with high-risk infants who showed signs of early language delay (n = 41) as well as with high- and low-risk infants who did not have ASD or language delay (n = 255 and 143, respectively). Results indicated that diminished language skills were evident at 12 months in infants with ASD and infants with early language delay. At 24 months of age, only the infants with ASD displayed atypical receptive-expressive language profiles. Associations between 12-month subcortical volumes and 24-month language skills were moderated by group status, indicating disordinal brain-behavior associations among infants with ASD and infants with language delay. These results suggest that there are different brain mechanisms influencing language development in infants with ASD and infants with language delay, and that the two groups likely experience unique sets of genetic and environmental risk factors.