Recent genome-sequencing studies have revealed dozens of genes frequently mutated in esophageal squamous cell carcinoma, but few genes are associated with patients' clinical outcomes. Novel prognostic biomarkers are urgently needed in the clinic. We collected both somatic mutations and clinical information of 442 Chinese esophageal squamous cell carcinoma patients from four published studies. Survival analysis was performed to reveal the clinical significance of the mutated genes. Dysregulation of the mutated genes was observed from public gene-expression data sets and its effects on cell migration and invasion were investigated with siRNA-mediated silencing. Our integrated analysis revealed 26 genes significantly and frequently mutated in esophageal squamous cell carcinoma. Importantly, mutations in ZFHX4, SPHKAP, NRXN1, KIAA1109, DNAH5 and KCNH7 were associated with poor survival. In addition, ZFHX4 was overexpressed in tumor tissues compared to normal controls, and knockdown of ZFHX4 in vitro significantly inhibited cell migration and invasion. Mutations in ZFHX4 were strongly associated with poor prognosis and the down-regulation of ZFHX4 inhibits the progression of esophageal squamous cell carcinoma. Further investigation is warranted to confirm the prognostic values of ZFHX4 in a prospective study.