By using transitionless quantum driving algorithm (TQDA), we present an efficient scheme for the shortcuts to the holonomic quantum computation (HQC). It works in decoherence-free subspace (DFS) and the adiabatic process can be speeded up in the shortest possible time. More interestingly, we give a physical implementation for our shortcuts to HQC with nitrogen-vacancy centers in diamonds dispersively coupled to a whispering-gallery mode microsphere cavity. It can be efficiently realized by controlling appropriately the frequencies of the external laser pulses. Also, our scheme has good scalability with more qubits. Different from previous works, we first use TQDA to realize a universal HQC in DFS, including not only two noncommuting accelerated single-qubit holonomic gates but also a accelerated two-qubit holonomic controlled-phase gate, which provides the necessary shortcuts for the complete set of gates required for universal quantum computation. Moreover, our experimentally realizable shortcuts require only two-body interactions, not four-body ones, and they work in the dispersive regime, which relax greatly the difficulty of their physical implementation in experiment. Our numerical calculations show that the present scheme is robust against decoherence with current experimental parameters.