Hybrid immunity induced by vaccination following recovery from SARS-CoV-2 infection is more robust than immunity induced by either infection or vaccination alone. To investigate how infection severity influenced the strength and character of subsequent vaccine-induced humoral or cellular immune responses against SARS-CoV-2, we assessed humoral and cellular immune responses against SARS-CoV-2 following recovery from infection, vaccine dose 1 and vaccine dose 2 in 35 persons recovered from COVID-19. Persons with polymerase chain reaction or serologically confirmed SARS-CoV-2 infection were recruited into a study of immunity against SARS-CoV-2. Self-reported symptoms categorized them as experiencing asymptomatic, mild, moderate or severe infection based on duration, intensity and need for hospitalization. Whole blood was obtained before vaccination and after first and second doses. Humoral immunity was assessed by ELISA and cellular immunity by ELISpot and intracellular flow cytometry. Responses were compared between groups recovered from either asymptomatic/mild (n = 14) or moderate/severe (n = 21) infection. Most subjects experienced robust increases in humoral and cellular immunity against SARS-CoV-2 spike (S) protein following 1 vaccination. Quantitative responses to second vaccination were marginal when measured 2.5 months afterwards and moderate or severe infection maintained stronger responses. Polyfunctional CD8⁺ T cell responses were largely restricted to subjects recovered from moderate or severe infection. One vaccine dose triggered stronger immune responses than in a comparable group never infected with SARS-CoV-2, while the second dose produced only minor lasting increases in humoral or cellular responses. Infection history should be considered in planning COVID-19 vaccine administration.