Nearly all chronic human infections are associated with alterations in the memory B cell (MBC) compartment, including a large expansion of CD19 T-bet MBC in the peripheral blood of HIV-infected individuals with chronic viremia. Despite their prevalence, it is unclear how these B cells arise and whether they contribute to the inefficiency of antibody-mediated immunity in chronic infectious diseases. We addressed these questions by characterizing T-bet-expressing B cells in lymph nodes (LN) and identifying a strong T-bet signature among HIV-specific MBC associated with poor immunologic outcome. Confocal microscopy and quantitative imaging revealed that T-bet B cells in LN of HIV-infected chronically viremic individuals distinctly accumulated outside germinal centers (GC), which are critical for optimal antibody responses. In single-cell analyses, LN T-bet B cells of HIV-infected individuals were almost exclusively found among CD19 MBC and expressed reduced GC-homing receptors. Furthermore, HIV-specific B cells of infected individuals were enriched among LN CD19 T-bet MBC and displayed a distinct transcriptome, with features similar to CD19 T-bet MBC in blood and LN GC B cells (GCBC). LN CD19 T-bet MBC were also related to GCBC by B cell receptor (BCR)-based phylogenetic linkage but had lower BCR mutation frequencies and reduced HIV-neutralizing capacity, consistent with diminished participation in GC-mediated affinity selection. Thus, in the setting of chronic immune activation associated with HIV viremia, failure of HIV-specific B cells to enter or remain in GC may help explain the rarity of high-affinity protective antibodies.