The extraordinary enhancement in heat transfer efficiency of nanofluids at extremely low volume fractions has attracted a lot of attention in identifying the governing mechanisms. The nanoscale effects, Brownian motion (random motion of particles inside the base fluid) and thermophoresis (diffusion of particles due to temperature gradient) are found to be important slip mechanisms in nanofluids. Based on these findings, a set of partial differential equations for conservation laws for nanofluids was formed. Since then, a large number of mathematical studies on convective heat transfer in nanofluids became feasible. The present paper summarizes the studies pertaining to instability of a horizontal nanofluid layer under the impact of various parameters such as rotation, magnetic field, Hall currents and LTNE effects in both porous and non-porous medium. Initially, investigations were made using the model considering fixed initial and boundary conditions on the layer, gradually the model was revised in the light of more practical boundary conditions and recently it has been modified to get new and more interesting results. The exhaustive analysis of instability problems is presented in the paper and prospects for future research are also identified.