Ergosterol peroxide (EP) in Anoectochilus elwesii possesses antioxidant and anticancer properties, yet few studies have been focused on its mechanisms and directed detecting. By practicing HPTLC-DPPH coupled with UHPLC-ESI-TOF MS, EP was located and the cytotoxic activity of EP was performed by MTT method. The apoptosis studies were conducted on SGC-7901cells by AO/EB and Annex V/PI staining method, PI flow cytometric assay, reactive oxygen species detection and mitochondrial membrane potential assay. An effect-directed detecting method of HPTLC-DPPH/UHPLC/ESI-TOF-MS was developed for EP rapidly and precisely, providing an option for identifying oxidative compounds. EP exhibits high cytotoxic activity against SGC-7901 gastric cancer cells and the morphological apoptosis suggested that EP induced apoptosis and cell cycle arrest in G0/G1 phase. It could enhance the ROS level and cause a decrease in the mitochondrial membrane potential. Its antiproliferative mechanism is G0/G1 phase arrest and might be traced through ROS-mediated mitochondrial dysfunction pathways.