Photocatalytic hydrogen evolution is a promising technique for the direct conversion of solar energy into chemical fuels. Colloidal quantum dots with tunable band gap and versatile surface properties remain among the most prominent targets in photocatalysis despite their frequent toxicity, which is detrimental for environmentally friendly technological implementations. In the present work, all-inorganic sulfide-capped InP and InP/ZnS quantum dots are introduced as competitive and far less toxic alternatives for photocatalytic hydrogen evolution in aqueous solution, reaching turnover numbers up to 128,000 based on quantum dots with a maximum internal quantum yield of 31%. In addition to the favorable band gap of InP quantum dots, in-depth studies show that the high efficiency also arises from successful ligand engineering with sulfide ions. Due to their small size and outstanding hole capture properties, sulfide ions effectively extract holes from quantum dots for exciton separation and decrease the physical and electrical barriers for charge transfer.