Online identification of the transient instability is essential in power system analysis, as it facilitates the grid operator to decide and coordinate system failure correction control actions. The rapid technological development of phasor measurement units (PMUs) provides an opportunity for online transient out-of-step identification in real-time. This paper analyzes the characteristics of angular velocity-acceleration trajectories in detail, and then proposes a predictive transient out-of-step identification method among coherent groups of generators based on the velocity-acceleration trajectory sampled by PMUs. To prove that the method can distinguish the transient out-of-step event following disturbance reliably, a mathematical proof of the criterion extended to multi-machine system is made, and the influences of the exciter and emergency control are taken into consideration to a certain extent. Finally, the effectiveness and predictability of the proposed method are demonstrated on a 39-bus New England test system and two large-scale power system simulation models of China by numerical studies.