Strong light-matter coupling can re-arrange the exciton energies in organic semiconductors. Here, we exploit strong coupling by embedding a fullerene-free organic solar cell (OSC) photo-active layer into an optical microcavity, leading to the formation of polariton peaks and a red-shift of the optical gap. At the same time, the open-circuit voltage of the device remains unaffected. This leads to reduced photon energy losses for the low-energy polaritons and a steepening of the absorption edge. While strong coupling reduces the optical gap, the energy of the charge-transfer state is not affected for large driving force donor-acceptor systems. Interestingly, this implies that strong coupling can be exploited in OSCs to reduce the driving force for electron transfer, without chemical or microstructural modifications of the photo-active layer. Our work demonstrates that the processes determining voltage losses in OSCs can now be tuned, and reduced to unprecedented values, simply by manipulating the device architecture.