•An up-to-date, comprehensive and compact overview of the vast amount of work on image and video based face recognition in the literature.•A novel taxonomy of image and video-based methods, which also contains recent methods such as sparsity and deep learning based methods.•An up-to-date review of the image and video-based data sets used for face recognition.•Review of the recent deep-learning based methods, which have shown remarkable results on large scale and unconstrained challenging data sets.•Information on both image and video-based methods, with an emphasis on the video-based methods. Biometric systems have the goal of measuring and analyzing the unique physical or behavioral characteristics of an individual. The main feature of biometric systems is the use of bodily structures with distinctive characteristics. In the literature, there are biometric systems that use physiological features (fingerprint, iris, palm print, face, etc.) as well as systems that use behavioral characteristics (signature, walking, speech patterns, facial dynamics, etc.) Recently, facial biometrics has been one of the most preferred biometric data since it generally does not require the cooperation of the user and can be obtained without violating the personal private space. In this paper, the methods used to obtain and classify facial biometric data in the literature have been summarized. We give a taxonomy of image-based and video-based face recognition methods, outline the major historical developments, and the main processing steps. Popular data sets that have been used for face recognition by researchers are also reviewed. We also cover the recent deep-learning based methods for face recognition and point out possible directions for future research.