A new iron(III) complex (1) of 5-nitro-8-hydroxylquinoline (HNOQ) was synthesized and structurally characterized in its solid state and solution state by IR, UV-Vis, electrospray ionization (ESI)-MS, elemental analysis, conductivity and X-ray single crystal diffraction analysis. The DNA binding study suggested that complex 1 interacted with calf thymus (ct)-DNA mainly via an intercalative binding mode. By 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide (MTT) assay, the in vitro cytotoxicity of complex 1, comparing with HNOQ and cisplatin, was screened towards a series of tumor cell lines as well as the normal liver cell line HL-7702. Complex 1 showed higher cytotoxicity towards the tested tumor cell lines but lower cytotoxicity towards HL-7702 than HNOQ, in which the T-24 was the most sensitive cell line for 1. Complex 1 caused G2 phase cell cycle arrest and induced cell apoptosis in T-24 cells in a dose-dependent mode, evidenced by changes in cell morphology. Targeting the mitochondrial pathway due to the redox potential of Fe(III)/Fe(II), the apoptotic mechanism in T-24 cells treated by 1 was investigated by reactive oxygen species (ROS) detection, intracellular Ca2+ measurement and caspase-9 and caspase-3 activity assay. It suggested that complex 1 induced cell apoptosis by triggering the caspase-9 and caspase-3 activation via a mitochondrion-mediated pathway.