Heavy metal pollution in soil has received much attention in recent decades. Many studies have analyzed the contamination status, spatial distribution, and pollution sources of heavy metals. Little information is available on the interaction between cultivated land quality and soil heavy metal pollution. Combining soil quality information and intensive heavy metal sampling surveys, this study analyzed heavy metal contamination and the ecological and health risks of various soils with different quality levels. Additionally, through the PMF model and risk assessment techniques, the ecological and health risks of specific pollution sources and their interaction with soil quality were investigated. The results showed that the mean content of the studied elements followed the increasing order of Hg (0.12 mg/kg) < Cd (0.19 mg/kg) < As (6.98 mg/kg) < Pb (25.57 mg/kg) < Cr (72.02 mg/kg). In addition, with increasing soil quality, the concentrations of Pb, Cr and Hg as well as the overall ecological risk increased significantly. Regarding health risks, heavy metal pollution posed a higher risk to children than adults, and ingestion was the main exposure pathway. The total hazard index and carcinogenic risk also increased with increasing soil quality. The PMF analysis showed that Pb and Cr mainly came from industrial activities, As could be attributed to natural sources, Cd was mainly derived from agricultural activities, and Hg pollution was determined by coal combustion. Considering the risks of specific pollution sources, agricultural activities and coal combustion were the major reasons for high ecological risks, whereas industrial activities and coal combustion posed significantly higher risks in suburban high-quality soil. Industrial activities mainly determined the health risk, which contributed more than 50% to the total risk. There was an upward health risk trend with increasing soil quality. Industrial activities in high-quality suburban soil posed the highest health risk to both adults and children. Reasonable and effective policies should be formulated to control industrial pollution and improve the ecological environment in this area.