Analysis of the specificity and kinetics of neutralizing antibodies (nAbs) elicited by SARS-CoV-2 infection is crucial for understanding immune protection and identifying targets for vaccine design. In a cohort of 647 SARS-CoV-2-infected subjects, we found that both the magnitude of Ab responses to SARS-CoV-2 spike (S) and nucleoprotein and nAb titers correlate with clinical scores. The receptor-binding domain (RBD) is immunodominant and the target of 90% of the neutralizing activity present in SARS-CoV-2 immune sera. Whereas overall RBD-specific serum IgG titers waned with a half-life of 49 days, nAb titers and avidity increased over time for some individuals, consistent with affinity maturation. We structurally defined an RBD antigenic map and serologically quantified serum Abs specific for distinct RBD epitopes leading to the identification of two major receptor-binding motif antigenic sites. Our results explain the immunodominance of the receptor-binding motif and will guide the design of COVID-19 vaccines and therapeutics. Display omitted •SARS-CoV-2 RBD is immunodominant and accounts for 90% of serum neutralizing activity•RBD antibodies decline with a half-life of ∼50 days, but their avidity increases•Structural definition of a SARS-CoV-2 RBD antigenic map using monoclonal antibodies•ACE2-binding site dominates SARS-CoV-2 polyclonal neutralizing antibody responses Serological analyses of ∼650 SARS-CoV-2-exposed individuals show that 90% of the serum or plasma neutralizing activity targets the virus receptor-binding domain, with structural insights revealing how distinct types of neutralizing antibodies targeting the ACE2-binding site dominate the immune response against SARS-CoV-2 spike.