Raman spectra of irradiated with fast neutrons or MeV ion-implanted radiation-damaged natural and CVD diamonds and chemically purified detonation nanodiamonds are investigated. The influence of radiation damage level and effects of high-temperature annealing on the intensity and spectral shape of the 1640 cm−1 band is studied. It is shown that in radiation-damaged diamonds this band consists of at least six Gaussian peaks, the intensity of which varies one to one both with the level of radiation disordering and the temperature of the subsequent annealing. The "1640" band in radiation-damaged diamonds is completely annealed at temperatures above 1000 °C, while in detonation nanodiamonds annealing up to 1200 °C does not significantly affect its shape and intensity.