Cancer-associated fibroblasts (CAFs), a stromal cell population with cell-of-origin, phenotypic and functional heterogeneity, are the most essential components of the tumor microenvironment (TME). Through multiple pathways, activated CAFs can promote tumor growth, angiogenesis, invasion and metastasis, along with extracellular matrix (ECM) remodeling and even chemoresistance. Numerous previous studies have confirmed the critical role of the interaction between CAFs and tumor cells in tumorigenesis and development. However, recently, the mutual effects of CAFs and the tumor immune microenvironment (TIME) have been identified as another key factor in promoting tumor progression. The TIME mainly consists of distinct immune cell populations in tumor islets and is highly associated with the antitumor immunological state in the TME. CAFs interact with tumor-infiltrating immune cells as well as other immune components within the TIME via the secretion of various cytokines, growth factors, chemokines, exosomes and other effector molecules, consequently shaping an immunosuppressive TME that enables cancer cells to evade surveillance of the immune system. In-depth studies of CAFs and immune microenvironment interactions, particularly the complicated mechanisms connecting CAFs with immune cells, might provide novel strategies for subsequent targeted immunotherapies. Herein, we shed light on recent advances regarding the direct and indirect crosstalk between CAFs and infiltrating immune cells and further summarize the possible immunoinhibitory mechanisms induced by CAFs in the TME. In addition, we present current related CAF-targeting immunotherapies and briefly describe some future perspectives on CAF research in the end.