Immunotherapy with anti-PD-1 antibody preliminarily showed promising efficacy for treating esophageal squamous cell carcinoma (ESCC). Herein, we used tissue microarrays and immunohistochemically analyzed PD-L1 and various tumor infiltrating immune cells (TIICs) in specimens from 196 ESCC patients who had undergone curative resection without preoperative therapy. PD-L1 expressions in tumor cells (TCs) and TIICs, as well as infiltration of lymphocytes (CD4+, CD8+, FOXP3+, and PD- 1+) and macrophages (CD68+ and CD204+), were evaluated. PD-L1 was expressed in TCs of 18.4% and in TIICs of 83.3% of these patients. PD-L1 expressions in TCs and TIICs were associated with significant infiltration of various TIIC types, especially CD8+ cells. PD-L1 expressions in both TCs and TIICs were significantly associated with favorable overall survival, and combining their levels enhanced prognostic accuracy. Prognostic impacts of PD-L1 expressions in TCs and TIICs, abundant PD-1+ cell infiltration, a high CD8+/FOXP3+ ratio, and the CD8+/CD204+ ratio remained significant after adjusting for clinicopathological factors. In conclusion, PD-L1 expression reflects anti-tumor immunity, and PD-1/PD-L1 expression and the ratio of infiltrating effector to immune suppressor cells have prognostic value. Therapeutic strategies inhibiting the PD-1/PD-L1 signal and immune suppressor cells are anticipated in ESCC patients.