We demonstrate that polymer electron acceptors with excellent all‐polymer solar‐cell (all‐PSC) device performance can be developed from polymer electron donors by using B←N units. By alleviating the steric hindrance effect of the bulky pendant moieties on the conjugated polymers that contain B←N units, the π–π stacking distance of polymer backbones is decreased and the electron mobility is consequently enhanced by nearly two orders of magnitude. As a result, the power conversion efficiency of all‐PSCs with the polymer acting as the electron acceptor is greatly improved from 0.12 % to 5.04 %. This PCE value is comparable to that of the best all‐PSCs with state‐of‐the‐art polymer acceptors. From giver to taker: Incorporation of B←N units into polymer electron donors has resulted in a series of polymer electron acceptors. Extending the length of the repeating units of the conjugated polymers alleviates the effect of steric hindrance from the pendant groups and promotes the π–π stacking of the polymer backbones. The all‐polymer solar‐cell device shows a power conversion efficiency (PCE) exceeding 5.0 %.