The splitting field of a matrix associated with a graph is the smallest field extension of Q that contains all of its eigenvalues. The extension degree is called its algebraic degree. In this paper, by introducing a new characteristic vector for each normal subset of a finite group, we completely determine the splitting fields and algebraic degrees for the adjacency matrix and distance matrix of a normal Cayley graph, which generalize the main results of Godsil et al. and Lu et al. Moreover, we study the relation between the algebraic integrality of these two matrices, and generalize a main result of Huang and Li. Finally, for a normal mixed Cayley graph, we consider its Hermitian adjacency matrix and Hermitian adjacency matrix of the second kind, and we characterize their splitting fields and algebraic degrees, which generalize the main results of Huang et al. and Kadyan et al.