Doping is a quite useful strategy for probing the structure and properties of metal nanoclusters, but the effect of doping on the photodynamical properties is still not fully understood. Here, we reveal that the number of valence electrons plays a major role in determining the photodynamics of M1Au24(SR)18 nanoclusters. By carrying out temperature-dependent optical absorption, it is found that Cd doping enhances electron–phonon coupling while Hg doping does not significantly alter the coupling. Moreover, the relaxation dynamics of M1Au24(SR)180 (M = Hg/Cd) nanoclusters show similar features to that of the negatively charged Au25 nanocluster. Specifically, the 8-electron M1Au24 (M = Cd/Hg) nanoclusters show a long excited-state lifetime (50−200 ns) and a weak picosecond relaxation, similar to the case of the anionic Au25− nanocluster. On the other hand, the non-8-electron MAu24 (M = Pd/Pt) nanoclusters show much more significant picosecond relaxation and thus much shorter excited-state lifetimes, which resembles the case of neutral Au250. The picosecond relaxation in all six cases can be explained by core–shell charge transfer or relaxation to the surface trap state. These results are of great importance for fundamental understanding of the interplay between the valence electrons and the optical properties of metal nanoclusters.