Individuals who receive a third mRNA vaccine dose show enhanced protection against severe COVID-19, but little is known about the impact of breakthrough infections on memory responses. Here, we examine the memory antibodies that develop after a third or fourth antigenic exposure by Delta or Omicron BA.1 infection, respectively. A third exposure to antigen by Delta breakthrough increases the number of memory B cells that produce antibodies with comparable potency and breadth to a third mRNA vaccine dose. A fourth antigenic exposure with Omicron BA.1 infection increased variant-specific plasma antibody and memory B cell responses. However, the fourth exposure did not increase the overall frequency of memory B cells or their general potency or breadth compared to a third mRNA vaccine dose. In conclusion, a third antigenic exposure by Delta infection elicits strain-specific memory responses and increases in the overall potency and breadth of the memory B cells. In contrast, the effects of a fourth antigenic exposure with Omicron BA.1 are limited to increased strain-specific memory with little effect on the potency or breadth of memory B cell antibodies. The results suggest that the effect of strain-specific boosting on memory B cell compartment may be limited.