•RCHFDE is proposed to extract raw fault features of gear.•HHO is adopted to determine the optimal parameters of DDMA and KELM.•A novel fault diagnosis model based on RCHFDE, HHO-DDMA&KELM is proposed.•The gear diagnosis case demonstrates the validity and superiority of the model. Accurate judgment of gear working state is essential to the normal operation of mechanical equipment. To effectively extract the dynamic features representing the gear state from the vibration signals, this paper proposes refined composite hierarchical fluctuation dispersion entropy (RCHFDE), where the composite hierarchical decomposition is employed to replace the traditional hierarchical decomposition to improve the performance of HFDE. Combining RCHFDE and manifold learning, a new gear fault diagnosis method is proposed. Firstly, RCHFDE is used to extract the original fault features. After that, optimized discriminant diffusion maps analysis is adopted to map high-dimensional features to low-dimensional subsets. Finally, the low-dimensional features are input into optimized kernel extreme learning machine to identify different fault states of gear. The experimental results show that, compared with other contrastive methods, the proposed method enjoys better performance, which can effectively complete the determination of different gear fault states.