As the core technology of hybrid electric vehicles (HEVs), energy management strategy directly affects the fuel consumption of vehicles. This research proposes a novel reinforcement learning (RL)-based algorithm for energy management strategy of HEVs. Hierarchical structure is used in deep Q-learning algorithm (DQL-H) to get the optimal solution of energy management. Through this new RL method, we not only solve the problem of sparse reward in training process, but also achieve the optimal power distribution. In addition, as a kind of hierarchical algorithm, DQL-H can change the way of exploration of the vehicle environment and make it more effective. The experimental results show that the proposed DQL-H method realizes better training efficiency and lower fuel consumption, compared to other RL-based ones. •DQL-H trains each level independently and is more efficient than counterparts.•Sparse rewards can be overcome during the training process.•Substantial rewards can accelerate the speed of convergence.•DQL-H changes the way of exploring the vehicle environment.