Studies of human mobility in the past decade revealed a number of general scaling laws. However, to reproduce the scaling behaviors quantitatively at both the individual and population levels simultaneously remains to be an outstanding problem. Moreover, recent evidence suggests that spatial scales have a significant effect on human mobility, raising the need for formulating a universal model suited for human mobility at different levels and spatial scales. Here we develop a general model by combining memory effect and population-induced competition to enable accurate prediction of human mobility based on population distribution only. A variety of individual and collective mobility patterns such as scaling behaviors and trajectory motifs are accurately predicted for different countries and cities of diverse spatial scales. Our model establishes a universal underlying mechanism capable of explaining a variety of human mobility behaviors, and has significant applications for understanding many dynamical processes associated with human mobility.