•The influence of the different diffuser BTE profiles on the flow instability in RCP is explored.•The pressure pulsations are extremely asymmetrical in the circumferential direction.•Two high axial-vorticity magnitude regions are captured on the mid-span.•The appropriate diffuser BTE profile would also diminish Von Kármán vortices. The shedding flow structures from the diffuser blade trailing edge, usually known as Von Kármán vortices, are complicated and crucial to the safe operation of the nuclear reactor coolant pump (RCP), if the shedding frequency reaches the resonant frequency of the diffuser. In the present study, numerical investigation is conducted to analyze the effect of the diffuser blade trailing edge (BTE) profile on the flow instability in a nuclear reactor coolant pump. Five typical diffuser BTEs are analyzed including original trailing edge (OTE), circular trailing edge (CTE), suction surface trailing edge radius 45mm (STER 45), suction surface trailing edge radius 60mm (STER 60) and suction surface trailing edge radius 75mm (STER 75). Results show that by changing the diffuser BTE profile, the vortex shedding intensity from the trailing edge would be diminished, and unsteady flow structures in the spherical casing are more uniform with the well modified diffuser BTE profile. When adopting the cases of STER 45, STER 60 and STER 75, pressure pulsations decrease at the diffuser outlet, but increase at the right side of spherical casing wall. From axial-vorticity distribution, it is indicated that the appropriate BTE profile can effectively prevent flow separation and change evolution of separate flow especially near the discharge nozzle. Besides, it would also diminish Von Kármán vortices from the diffuser BTE, and improve the RCP hydraulic efficiency.