The past few years has shown a sudden spurt in the field of human computer interaction. The days of using a mouse to control a computer is almost obsolete and people now prefer to use touch screens and more recently, air gestures, for the same. However, the use of gestures is not limited to computers alone. It finds its application in controlling televisions and other home appliances as well. This paper explores one such application where in air gestures could be used to control automobiles. The paper describes a novel method to not only give directions but also password-protect and use special features of the vehicle, using gestures. The complete algorithm was developed using video processing on MATLAB 2011b and was found to be 3.6 times faster than its predecessor algorithms. The same was tested in real-time as well, using a robot prototype and satisfactory results were obtained.