Despite the overall clinical success of immune checkpoint inhibitors (ICIs) for treating patients with solid tumors, a large number of patients do not benefit from this approach. Consequently, there is a need for predictive biomarkers. The most prevalent biomarkers such as PD-L1 expression and tumor mutational burden (TMB) do not reliably predict response to ICIs across different solid tumor types suggesting that a broader view of regulating factors in the tumor microenvironment is needed. Emerging evidence indicates that one central common denominator of resistance to ICIs may be fibrotic activity characterized by extracellular matrix (ECM) and collagen production by cancer-associated fibroblasts (CAFs). A fibroblast-and collagen-rich stroma attenuates immunotherapy response by contributing to inhibition and exclusion of T cells. Here we review opportunities and limitations in the utilization of the most prevalent biomarkers for ICIs and elaborate on the unique opportunities with biomarkers originating from the activated fibroblasts producing an impermeable ECM. We propose that ECM and collagen biomarkers measured non-invasively may be a novel and practical approach to optimize treatment strategies and improve patient selection for ICI therapy.