Antibody titers against SARS-CoV-2 slowly wane over time. Here, we examined how time affects antibody potency. To assess the impact of antibody maturation on durable neutralizing activity against original SARS-CoV-2 and emerging variants of concern (VOCs), we analyzed receptor binding domain (RBD)-specific IgG antibodies in convalescent plasma taken 1–10 months after SARS-CoV-2 infection. Longitudinal evaluation of total RBD IgG and neutralizing antibody revealed declining total antibody titers but improved neutralization potency per antibody to original SARS-CoV-2, indicative of antibody response maturation. Neutralization assays with authentic viruses revealed that early antibodies capable of neutralizing original SARS-CoV-2 had limited reactivity toward B.1.351 (501Y.V2) and P.1 (501Y.V3) variants. Antibodies from late convalescents exhibited increased neutralization potency to VOCs, suggesting persistence of cross-neutralizing antibodies in plasma. Thus, maturation of the antibody response to SARS-CoV-2 potentiates cross-neutralizing ability to circulating variants, suggesting that declining antibody titers may not be indicative of declining protection. Display omitted •Qualitative changes in plasma neutralizing antibody are longitudinally analyzed•Affinity-matured antibodies with resistance to variants are durably maintained•Neutralizing potency and breadth to SARS-CoV-2 variants increase with time•Serological immunity evolves with time to counter SARS-CoV-2 variants Antigenic drifts in SARS-CoV-2 variants permit escape from neutralizing antibody in COVID-19 convalescent plasma. Moriyama et al. reveal the evolution of serological immunity with time that counters SARS-CoV-2 variants via affinity maturation and durable elicitation of IgG antibodies that are resistant to viral escape.