Knowledge of immune cell phenotypes in the tumor microenvironment is essential for understanding mechanisms of cancer progression and immunotherapy response. We profiled 45,000 immune cells from eight breast carcinomas, as well as matched normal breast tissue, blood, and lymph nodes, using single-cell RNA-seq. We developed a preprocessing pipeline, SEQC, and a Bayesian clustering and normalization method, Biscuit, to address computational challenges inherent to single-cell data. Despite significant similarity between normal and tumor tissue-resident immune cells, we observed continuous phenotypic expansions specific to the tumor microenvironment. Analysis of paired single-cell RNA and T cell receptor (TCR) sequencing data from 27,000 additional T cells revealed the combinatorial impact of TCR utilization on phenotypic diversity. Our results support a model of continuous activation in T cells and do not comport with the macrophage polarization model in cancer. Our results have important implications for characterizing tumor-infiltrating immune cells. Display omitted •Single-cell RNA-seq reveals phenotypic expansion of intratumoral immune cells•Biscuit identifies cell populations that differ in co-expression patterns•T cells reside on continuous activation and differentiation trajectories•Combinatorial environmental inputs and TCR usage shape T cell phenotypes Single-cell analysis of the breast tumor immune microenvironment, coupled with computational analysis, yields an immune map of breast cancer that points to continuous T cell activation and differentiation states.