Pure high impact polystyrene (HIPS) is widely used, but it is flammable. During combustion, it produces black smoke, drips, and poses other hazards to human safety, making it necessary to study flame-retardant HIPS. In this study, nitrogen-phosphorus-silicon flame retardants were prepared by using melamine cyanurate (MC), aluminum hypophosphite (ALHP), and nano-silica (nano-SiO2) to improve the flame retardancy of HIPS. The flame retardancy and mechanism of HIPS composites were studied. The results showed that when the ratio of MC to ALHP was 1:4 and nano-SiO2 was 1.5 wt%, the limiting oxygen index (LOI) value of HIPS composite was 26.9%, and PHRR and THR were reduced to 254 kW/m2 and 85 MJ/m2, respectively. The smoke production decreased by 32%, and the carbon residue increased by 70%. ALHP and nano-SiO2 mainly play a flame retardant role in the condensed phase. MC mainly plays a flame retardant role in the gas phase, as well as the condensed phase. Display omitted •The filling amount was lower than single silicon flame retardant.•The efficiency was higher than single nitrogen flame retardant.•The carbonization effect was better than single phosphorus flame retardant.•MC, ALHP, and nano-SiO2 were compounded and added to HIPS for the first time.•The flame retardant composite is an environmentally friendly material.