Programmed cell death protein 1 (PD-1) axis blockade has become the mainstay in the treatment of recurrent and/or metastatic (R/M) head and neck squamous cell cancer (HNSCC). Programmed death-ligand 1 (PD-L1) is the only approved biomarker for patient selection; however, response rate is limited even among high expressors. Our primary objective was to investigate the association of immune cell-related biomarkers in the tumor and tumor microenvironment with PD-1 checkpoint inhibitors’ outcomes in patients with R/M HNSCC. NCT03652142 was a prospective study in nivolumab-treated platinum-refractory R/M HNSCC, aiming to evaluate biomarkers of response to treatment. Tumor biopsies and blood samples were collected from 60 patients at baseline, post-treatment, and at progression. Immune cells in the tumor and stromal compartments were quantified by immunofluorescence using a five-protein panel (CD3, CD8, CD20, FoxP3, cytokeratin). Tertiary lymphoid structures (TLSs), PD-L1 expression, and peripheral blood immune cell composition were also evaluated for associations with outcome. Our findings were validated by gene set enrichment analysis (GSEA) messenger RNA in situ expression data from the same patients, for B-cell- and TLS-associated genes. High pre-treatment density of stromal B cells was associated with prolonged progression-free survival (PFS) (P = 0.011). This result was validated by GSEA, as stromal enrichment with B-cell-associated genes showed association with response to nivolumab. PD-L1 positivity combined with high B-cell counts in stroma defined a subgroup with significantly longer PFS and overall survival (P = 0.013 and P = 0.0028, respectively). Increased B cells in pre-treatment HNSCC biopsy samples correlate with prolonged benefit from PD-1-based immunotherapy and could further enhance the predictive value of PD-L1 expression. •PD-1/PD-L1 axis inhibitors have become the standard of care for R/M HNSCC.•Low response rates dictate the need to identify resistance mechanisms and develop biomarkers for patient selection.•The associations of immune cell density with treatment outcomes were investigated in nivolumab-treated R/M HNSCC cases.•Increased B-cell infiltration led to prolonged PFS and improved PD-L1-based patient selection.•The synchronous assessment of B-cell infiltration and PD-L1 expression could guide therapeutic decisions in R/M HNSCC.