The spike glycoprotein of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) continues to accumulate substitutions, leading to breakthrough infections of vaccinated individuals. It remains unclear if exposures to antigenically distant SARS-CoV-2 variants can overcome memory B cell biases established by initial SARS-CoV-2 encounters. We determined the specificity and functionality of antibody and B cell responses following exposure to BA.5 and XBB variants in individuals who received ancestral SARS-CoV-2 mRNA vaccines. BA.5 exposures elicited antibody responses that targeted epitopes conserved between the BA.5 and ancestral spike. XBB exposures also elicited antibody responses that primarily targeted epitopes conserved between the XBB.1.5 and ancestral spike. However, unlike BA.5, a single XBB exposure elicited low frequencies of XBB.1.5-specific antibodies and B cells in some individuals. Pre-existing cross-reactive B cells and antibodies were correlated with stronger overall responses to XBB but weaker XBB-specific responses, suggesting that baseline immunity influences the activation of variant-specific SARS-CoV-2 responses. Display omitted •Variant breakthrough infections boost ancestral cross-reactive antibodies and B cells•First and second BA.5 exposures fail to elicit variant-specific antibodies and B cells•XBB infections and vaccinations elicit XBB-specific responses in some individuals•XBB-specific responses correlate with low levels of pre-existing humoral immunity It is unknown if antigenically distant SARS-CoV-2 variants can overcome memory B cell biases established by initial SARS-CoV-2 encounters. Johnston, Painter, Li et al. show that BA.5 and XBB exposures recall B cells targeting conserved epitopes in the ancestral SARS-CoV-2 spike protein. They found instances of XBB-specific responses after XBB exposures, which correlated with low baseline levels of humoral immunity.