This study aimed to investigate the correlation between primary tumor volume and cancer failure patterns in esophageal squamous cell carcinoma (ESCC) treated with definitive concurrent chemoradiotherapy (CCRT) and examine whether increasing radiation dose can improve the outcome. We retrospectively reviewed 124 patients with stage III ESCC treated by definitive CCRT. The primary tumor volume calculated from the radiotherapy planning computed tomography scans was correlated to treatment response, time to disease progression, and overall survival. We further analyzed whether a higher radiation dose correlated with better disease control and patient survival. Patients with poor CCRT response had a larger primary tumor volume than those with good response (97.9 vs 64.3 cm3, P = 0.032). The optimal cutoff value to predict CCRT response was 55.3 cm3. Large primary tumor volume (≥ 55.3 cm3) correlated with shorter time to tumor progression in the esophagus (13.6 vs 48.6 months, P = 0.033) compared with small tumor volume (< 55.3 cm3). For the large esophageal tumors (≥ 55.3 cm3), radiation dose > 60 gray significantly prolonged the time to tumor progression in esophagus (20.3 vs 10.1 months, P = 0.036) and overall survival (12.2 vs 8.0 months, P = 0.030), compared with dose ≤ 60 gray. In contrast, higher radiation dose did not benefit local disease control or overall survival in the small esophageal tumors (< 55.3 cm3). Large primary tumor volume correlates with poor local control and overall survival in ESCC treated with definitive CCRT. Radiation dose > 60 gray can improve the outcomes in patients with large primary tumor. Further prospective dose escalation trials are warranted.