Understanding adaptive immunity to SARS-CoV-2 is important for vaccine development, interpreting coronavirus disease 2019 (COVID-19) pathogenesis, and calibration of pandemic control measures. Using HLA class I and II predicted peptide “megapools,” circulating SARS-CoV-2-specific CD8+ and CD4+ T cells were identified in ∼70% and 100% of COVID-19 convalescent patients, respectively. CD4+ T cell responses to spike, the main target of most vaccine efforts, were robust and correlated with the magnitude of the anti-SARS-CoV-2 IgG and IgA titers. The M, spike, and N proteins each accounted for 11%–27% of the total CD4+ response, with additional responses commonly targeting nsp3, nsp4, ORF3a, and ORF8, among others. For CD8+ T cells, spike and M were recognized, with at least eight SARS-CoV-2 ORFs targeted. Importantly, we detected SARS-CoV-2-reactive CD4+ T cells in ∼40%–60% of unexposed individuals, suggesting cross-reactive T cell recognition between circulating “common cold” coronaviruses and SARS-CoV-2. Display omitted •Measuring immunity to SARS-CoV-2 is key for understanding COVID-19 and vaccine development•Epitope pools detect CD4+ and CD8+ T cells in 100% and 70% of convalescent COVID patients•T cell responses are focused not only on spike but also on M, N, and other ORFs•T cell reactivity to SARS-CoV-2 epitopes is also detected in non-exposed individuals An analysis of immune cell responses to SARS-CoV-2 from recovered patients identifies the regions of the virus that is targeted and also reveals cross-reactivity with other common circulating coronaviruses