The high-mannose patch on the HIV-1 envelope (Env) glycoprotein is the epicenter for binding of the potent broadly neutralizing PGT121 family of antibodies, but strategies for generating such antibodies by vaccination have not been defined. We generated structures of inferred antibody intermediates by X-ray crystallography and electron microscopy to elucidate the molecular events that occurred during evolution of this family. Binding analyses revealed that affinity maturation was primarily focused on avoiding, accommodating, or binding the N137 glycan. The overall antibody approach angle to Env was defined very early in the maturation process, yet some variation evolved in the PGT121 family branches that led to differences in glycan specificities in their respective epitopes. Furthermore, we determined a crystal structure of the recombinant BG505 SOSIP.664 HIV-1 trimer with a PGT121 family member at 3.0 Å that, in concert with these antibody intermediate structures, provides insights to advance design of HIV vaccine candidates. Display omitted •X-ray and EM structures of inferred precursors of the PGT121 family were generated•The N137 glycan is a major factor in affinity maturation of the PGT121 family•The antibody approach angle differs in the two main branches of the PGT121 lineage•A 3.0 Å crystal structure of an HIV trimer with a PGT121 family member was determined Strategies for generating broadly neutralizing antibodies (bnAbs) against the high-mannose patch of the HIV envelope have not been defined. Wilson and colleagues find that affinity maturation of the potent PGT121 family of bnAbs is focused on accommodating or avoiding glycans. These findings provide insights to advance structure-based vaccine design.