•A fractional-order hyperchaotic complex system is constructed.•The fractional-order hyperchaotic complex system is digital circuit implemented by DSP platform.•An image encryption algorithm based on fractional-order hyperchaotic complex system is proposed.•Simulation results and performance analysis demonstrate the security and robustness of the proposed algorithm. To explore characteristic of a new system with fractional order and complex variables, based on Adomian decomposition method (ADM), its nonlinear dynamical behaviors are discussed by bifurcation diagram, Lyapunov exponent spectrum, spectral entropy (SE) complexity, and DSP implementation. It is found that, for each parameter and order, there exists a large interval to make the system remain a hyper-chaotic state, which illustrates that the hyper-chaotic sequence could be more random to be applied in image encryption. Especially, a novel image encryption algorithm is proposed based on the fractional-order hyper-chaotic complex system and Galois field (GF). The column cyclic shift is used to scrambling part, the GF(17) algorithm is to diffuse pixel value. The security performances of algorithm are also analyzed. Finally, experiment results indicate that the proposed algorithm not only can effectively encrypt image, but also has better security features.