Programmed cell death protein 1 (PD-1) inhibitors have efficacy in treating squamous cell carcinoma of the head and neck (SCCHN), but objective response rates are low. PD-1 ligand (PD-L1) expression alone is not considered a robust predictor of response and additional biomarkers are needed. This 3-year observational cohort followed 126 SCCHN patients treated with anti-PD-1/L1 therapy. Prior to treatment, 81 (64%) had targeted massively parallel tumor sequencing. Of these, 42 (52%) underwent fluorescence-activated cell sorting and PD-L1 immunohistochemistry for tumor immunoprofiling. Six (5%) complete responses (CRs) and 11 (9%) partial responses (PRs) were observed. Those treated with prior chemotherapy (98, 78%) versus only surgery and/or radiation had longer overall survival (OS) (10 vs. 3 months, P = 0.02). Smokers had a higher total mutational burden (TMB) (P = 0.01). Virus-positive patients had a lower TMB (P < 0.01) and improved OS (P = 0.02). Among virus-negative responders, NOTCH1 and SMARCA4 were more frequently mutated and frameshift events in tumor suppressor genes occurred more frequently (P = 0.03). Higher TMB and CD8+ T cell infiltrates predicted anti-PD-1/L1 benefit (P < 0.01, P < 0.01, respectively) among virus-negative tumors. TIM-3/LAG-3 coexpression with PD-1 was higher on T cells among nonresponders (P = 0.03 and 0.02, respectively). Somatic frameshift events in tumor suppressor genes and higher TMB among virus-negative SCCHN tumors predict anti-PD-1/L1 response.