This article studies the robust trajectory tracking control problem of a quadrotor unmanned aerial vehicle (UAV). In order to guarantee the desired trajectory tracking performance in the presence of external disturbances and model uncertainties, the design process of the quadrotor UAV controller is divided into two steps. First, by decomposing the attitude dynamic system into two serial-connected subsystems, a cascade active disturbance rejection control scheme is applied to the attitude subsystem. Second, by introducing an additional high-gain design parameter, a novel backstepping sliding-mode control scheme for position subsystem is constructed. Moreover, the Lyapunov stability analysis is provided to show that the trajectory tracking error can converge to an arbitrarily small residual set. Numerical results illustrate the effectiveness of the designed control method and its robustness to the external disturbances and model uncertainties. Finally, the proposed method is implemented on a quadrotor UAV to demonstrate its feasibility in practical application.