The artificial intelligence (AI) techniques have been widely used in the transient stability analysis of a power system. They are recognized as the most promising approaches for predicting the post-fault transient stability status with the use of phasor measurement units data. However, the popular AI methods used for power systems are often "black boxes," which result in the poor interpretation of the model. In this paper, a transient stability prediction method based on extreme gradient boosting is proposed. In this model, a decision graph and feature importance scores are provided to discover the relationship between the features of the power system and transient stability. Meanwhile, the key features are selected according to the feature importance scores to remove redundant variables. The simulation results on the New England 39-bus system have demonstrated the superiority of the proposed model over the prior methods in the computation speed and prediction accuracy. Finally, an algorithm is proposed to interpret the prediction results for a specific fault of the power system, which further improves the interpretability of the model and makes it attractive for real-time transient stability prediction.