The level of selenocysteine (Sec) in the human body is closely related to a variety of pathophysiological states, so it is important to study its fluorescence sensing mechanism for designing efficient fluorescent probes. Herein, we used time-dependent density functional theory to investigate the fluorescence sensing mechanism of phenanthroimidazole derivates A4 and B4 for the detection of Sec, which are proposed to be designed based on excited state intramolecular proton transfer (ESIPT) and intramolecular charge transfer (ICT) mechanisms. The calculation results show that the fluorescence quenching mechanism of A4 and B4 is due to the photo-induced electron transfer (PET) process with the sulfonate group acts as the electron acceptor. Subsequently, A4 and B4 react with Sec, the sulfonate group is substituted by hydroxyl groups, PET is turned off, and significant fluorescence enhancement of the formed A3 and B3 is observed. The theoretical results suggest that the fluorescence enhancement mechanism of B3 is not based on ICT mechanism, and the charge transfer phenomenon was not observed by calculating the frontier molecular orbitals, and proved to be a local excitation mode. The reason for the fluorescence enhancement of A3 based on ESIPT is also explained by the calculated potential energy curves.