To simultaneously achieve low photon energy loss (E loss) and broad spectral response, the molecular design of the wide band gap (WBG) donor polymer with a deep HOMO level is of critical importance in fullerene-free polymer solar cells (PSCs). Herein, we developed a new benzodithiophene unit, i.e., DTBDT-EF, and conducted systematic investigations on a WBG DTBDT-EF-based donor polymer, namely, PDTB-EF-T. Due to the synergistic electron-withdrawing effect of the fluorine atom and ester group, PDTB-EF-T exhibits a higher oxidation potential, i.e., a deeper HOMO level (ca. −5.5 eV) than most well-known donor polymers. Hence, a high open-circuit voltage of 0.90 V was obtained when paired with a fluorinated small molecule acceptor (IT-4F), corresponding to a low E loss of 0.62 eV. Furthermore, side-chain engineering demonstrated that subtle side-chain modulation of the ester greatly influences the aggregation effects and molecular packing of polymer PDTB-EF-T. With the benefits of the stronger interchain π–π interaction, the improved ordering structure, and thus the highest hole mobility, the most symmetric charge transport and reduced recombination are achieved for the linear decyl-substituted PDTB-EF-T (P2)-based PSCs, leading to the highest short-circuit current density and fill factor (FF). Due to the high Flory–Huggins interaction parameter (χ), surface-directed phase separation occurs in the P2:IT-4F blend, which is supported by X-ray photoemission spectroscopy results and cross-sectional transmission electron microscope images. By taking advantage of the vertical phase distribution of the P2:IT-4F blend, a high power conversion efficiency (PCE) of 14.2% with an outstanding FF of 0.76 was recorded for inverted devices. These results demonstrate the great potential of the DTBDT-EF unit for future organic photovoltaic applications.