Fructus Alpiniae oxyphyllae (A. oxyphylla) is a traditional herb which is widely used in East Asian for the treatment of dyspepsia, diarrhea, abdominal pain, poor memory, inflammatory conditions and cancer. The cytotoxic activities of ethanol extract (EE) and five extract layers including petroleum ether (PE), dichloromethane (DCLM), acetoacetate (EtOAc), n-Butanol (n-Bu) and water fractions (WF) of A. oxyphylla were tested on HepG2, SW480, MCF-7, K562 and HUVEC cell lines using MTT assay and LDH release assay. The component analysis was performed on HPLC with gradient elution. Hoechst 33342 staining, DCFH-DA fluorescence microscopy, flow cytometry analysis, western blot and migration assays were carried out to determine the anti-cancer mechanisms of PE. MTT analysis showed that EE, PE and DCLM could inhibit cell proliferation on HepG2, SW480, MCF-7, K562 and HUVEC cell lines, especially PE fraction. HPLC analysis pointed out five main components which may contribute to the anti-proliferative activity of PE. Further study showed that PE increased LDH release, induced apoptosis, disrupted mitochondrial membrane potential and elevated intracellular reactive oxygen species (ROS) in HepG2 cells, whereas the antioxidant N-acetyl-l-cysteine (NAC) prevented PE-induced ROS generation. The results of western blot revealed that PE induced apoptosis in HepG2 cells by enhancing Bax/Bcl-2 ratio, increasing cytochrome c in cytosol and activating caspase-3/9. Meanwhile, high levels of ROS could induce DNA damage-mediated protein expression, AKT, ERK inactivation and SAPKs activation. Furthermore, PE conspicuously blocked the migration of HUVEC cells. The present results demonstrated that PE induced apoptosis in HepG2 cells may be via a ROS-mediated signaling pathway. Display omitted