Colloidal quantum dots (QDs) have attracted scientific interest for infrared (IR) optoelectronic devices due to their bandgap tunability and the ease of fabrication on arbitrary substrates. In this work, short‐wave IR photodetectors based on lead sulfide (PbS) QDs with high detectivity and low dark current is demonstrated. Using a combination of time‐resolved photoluminescence, carrier transport, and capacitance–voltage measurements, it is proved that the charge carrier diffusion length in the QD layer is negligible such that only photogenerated charges in the space charge region can be collected. To maximize the carrier extraction, an optical model for PbS QD‐based photodiodes is developed, and through optical engineering, the cavity at the wavelength of choice is optimized. This universal optimization recipe is applied to detectors sensitive to wavelengths above 1.4 µm, leading to external quantum efficiency of 30% and specific detectivity (D*) in the range of 1012 Jones. An optical model for lead sulfide quantum dot thin films is developed and applied on short‐wave infrared sensitive photodetectors. Through optical engineering, the cavity is optimized at the wavelength of choice, leading to significant boost in the device external quantum efficiency, achieving values higher than 30% at wavelengths above 1.4 µm.