Zika virus (ZIKV), a mosquito-borne flavivirus with homology to Dengue virus (DENV), has become a public health emergency. By characterizing memory lymphocytes from ZIKV-infected patients, we dissected ZIKV-specific and DENV-cross-reactive immune responses. Antibodies to nonstructural protein 1 (NS1) were largely ZIKV-specific and were used to develop a serological diagnostic tool. In contrast, antibodies against E protein domain l/ll (EDI/II) were cross-reactive and, although poorly neutralizing, potently enhanced ZIKV and DENV infection in vitro and lethally enhanced DENV disease in mice. Memory T cells against NS1 or E proteins were poorly cross-reactive, even in donors preexposed to DENV. The most potent neutralizing antibodies were ZIKV-specific and targeted EDIII or quaternary epitopes on infectious virus. An EDIII-specific antibody protected mice from lethal ZIKV infection, illustrating the potential for antibody-based therapy.