The immunoprotective components control COVID-19 disease severity, as well as long-term adaptive immunity maintenance and subsequent reinfection risk discrepancies across initial COVID-19 severity, remain unclarified. Here, we longitudinally analyzed SARS-CoV-2-specific immune effectors during the acute infection and convalescent phases of 165 patients with COVID-19 categorized by severity. We found that early and robust SARS-CoV-2-specific CD4 and CD8 T cell responses ameliorate disease progression and shortened hospital stay, while delayed and attenuated virus-specific CD8 T cell responses are prominent severe COVID-19 features. Delayed antiviral antibody generation rather than titer level associates with severe outcomes. Conversely, initial COVID-19 severity imprints the long-term maintenance of SARS-CoV-2-specific adaptive immunity, demonstrating that severe convalescents exhibited more sustained virus-specific antibodies and memory T cell responses compared to mild/moderate counterparts. Moreover, initial COVID-19 severity inversely correlates with SARS-CoV-2 reinfection risk. Overall, our study unravels the complicated interaction between temporal characteristics of virus-specific T cell responses and COVID-19 severity to guide future SARS-CoV-2 wave management.