Learning-based face aging/rejuvenation has witnessed rapid progress in recent years. However, existing methods still suffer from the loss of personalized identity information when synthesizing cross-age faces. In this paper, we propose a Conditional Adversarial Consistent Identity AutoEncoder (CACIAE) to revisit this problem. Firstly, a Res-Encoder is designed to better generate powerful face representation. Secondly, the rectangular kernel is introduced into the encoder to make full use of horizontal continuous characteristic information of faces and to make the synthetic face images more natural. Thirdly, a novel consistent identity loss is proposed to learn more face details and produce more natural identity-preserving images. Further, two discriminators are designed to enforce the generator to generate more realistic and more age-accurate images. Experimental results prove the effectiveness of the proposed method, both qualitatively and quantitatively. The code is available at https://github.com/XH-B/CACIAE.