The reduced switch count quasi-Z-source three-level inverter (RSC-QZS-TLI) not only maintains the advantages of continuous input current in the conventional quasi-Z-source three-level inverter (C-QZS-TLI), but also reduces the number of power switches and system cost. However, it inevitably faces the challenges of high common-mode voltage (CMV) and neutral-point voltage (NPV) imbalance, which threaten system safety and reliability. Moreover, due to the inherent features of this topology, the medium vectors cannot be generated, which means that the traditional modulation method is not applicable any more. To address the above challenges, a novel modulation method is proposed. The pre-selected basic vectors are optimized primarily, which restrict the CMV magnitude within one-sixth of dc-link voltage. Then, for the particularity of RSC-QZS-TLI, the distribution factor of small vector and its limitation range are introduced for the first time, and an indirect calculating approach is developed to obtain the duty cycles of the pre-selected four vectors, which maintains the correct ac output voltage and dc voltage boosting ability. Meanwhile, according to sector number and output of NPV controller, the distribution factor is modified to control the NPV balance. Compared with the traditional modulation method, the CMV magnitude of the proposed method is reduced by 50 %, and the balanced NPV is controlled actively. Simulated and experimental results verify the feasibility of the proposed scheme.