Batch preparation of nano-HMX was achieved via a mechanical trituration method. The morphology and particle size of nano-HMX and raw RDX were characterized using SEM. Then nano-HMX was used in a formulation of composite modified double base propellant containing RDX. The method is to use nano- HMX to replace the RDX in the formulation by 10% gradually with the total mass content of RDX and HMX unchanged. The burning rate, mechanical sensitivity and mechanical property of propellant strands with different mass content of nano-HMX were tested. The results indicate that the 30% content of nano- HMX has the best comprehensive performance which can be used as an improvement of the existing formula. A possible mechanism of action was discussed.