Highly sensitive photodetection even approaching the single-photon level is critical to many important applications. Graphene-based hybrid phototransistors are particularly promising for high-sensitivity photodetection because they have high photoconductive gain due to the high mobility of graphene. Given their remarkable optoelectronic properties and solution-based processing, colloidal quantum dots (QDs) have been preferentially used to fabricate graphene-based hybrid phototransistors. However, the resulting QD/graphene hybrid phototransistors face the challenge of extending the photodetection into the technologically important mid-infrared (MIR) region. Here, we demonstrate the highly sensitive MIR photodetection of QD/graphene hybrid phototransistors by using plasmonic silicon (Si) QDs doped with boron (B). The localized surface plasmon resonance (LSPR) of B-doped Si QDs enhances the MIR absorption of graphene. The electron-transition-based optical absorption of B-doped Si QDs in the ultraviolet (UV) to near-infrared (NIR) region additionally leads to photogating for graphene. The resulting UV-to-MIR ultrabroadband photodetection of our QD/graphene hybrid phototransistors features ultrahigh responsivity (up to ∼109 A/W), gain (up to ∼1012), and specific detectivity (up to ∼1013 Jones).