Statistics have shown that many accidents occur due to drowsy condition of drivers. In a study conducted by National Sleep Foundation, it has been found that about 20% of drivers feel drowsy during driving. These statistics paint a very scary picture. This paper proposes a system for driver drowsiness detection, in which the architecture detects sleepiness of driver. The proposed architecture consists of four deep learning models: AlexNet, VGG-FaceNet, FlowImageNet and ResNet, which use RGB videos of drivers as input and help in detecting drowsiness. Also, these models consider four types of different features such as hand gestures, facial expressions, behavioral features and head movements for the implementation. The AlexNet model is used for various background and environmental changes like indoor, outdoor, day and night. VGG-FaceNet is used to extract facial characteristics like gender ethnicities. FlowImageNet is used for behavioral features and head gestures, and ResNet is used for hand gestures. Hand gestures detection provides a precise and accurate result. These models classify these features into four classes: non-drowsiness, drowsiness with eye blinking, yawning and nodding. The output of these models is provided to ensemble algorithm to obtain a final output by putting them through a SoftMax classifier that gives us a positive (drowsy) or negative answer. The accuracy obtained from this system came out to be 85%.