Trirutile NiTa2O6 has been studied under high pressure by in situ Raman and angle-dispersive synchrotron X-ray diffraction techniques. It undergoes a new quenchable phase at high pressures above 11.8 GPa accompanied by softening of the internal modes ν1(A1g), ν1(Eg), and ν6(Eg), and it is denser by 15% compared with its ambient phase. Various Raman-active modes of NiTa2O6 diminished at high pressures due to the distortion of edge-sharing TaO6 octahedra, which was further confirmed by X-ray diffraction and density functional theory results. The equation of state has been determined using the second-order Birch–Murnaghan equation, and the obtained bulk modulus is 199(4) GPa. The pressure and volume dependence of optical lattice vibrational frequencies and their corresponding Grüneisen parameters are calculated, indicating the inconsistency of the trirutile structure at high pressures, which was accompanied by the strong deformation of TaO6 octahedra. Pressure-induced structural metamorphosis and soft-mode-driven displacive transition related to the mechanical instability of NiTa2O6 are examined and decompression results recommend the transition is irreversible.