COVID-19 is a highly infectious respiratory disease caused by the novel coronavirus SARS-CoV-2. It has become a global pandemic and its frequent mutations may pose new challenges for vaccine design. During viral infection, the Spike RBD of SARS-CoV-2 binds the human host cell receptor ACE2, enabling the virus to enter the host cell. Both the Spike and ACE2 are densely glycosylated, and it is unclear how distinctive glycan types may modulate the interaction of RBD and ACE2. Detailed understanding of these determinants is key for the development of novel therapeutic strategies. To this end, we perform extensive all-atom simulations of the (i) RBD-ACE2 complex without glycans, (ii) RBD-ACE2 with oligomannose MAN9 glycans in ACE2, and (iii) RBD-ACE2 with complex FA2 glycans in ACE2. These simulations identify the key residues at the RBD-ACE2 interface that form contacts with higher probabilities, thus providing a quantitative evaluation that complements recent structural studies. Notably, we find that this RBD-ACE2 contact signature is not altered by the presence of different glycoforms, suggesting that RBD-ACE2 interaction is robust. Applying our simulated results, we illustrate how the recently prevalent N501Y mutation may alter specific interactions with host ACE2 that facilitate the virus-host binding. Furthermore, our simulations reveal how the glycan on Asn90 of ACE2 can play a distinct role in the binding and unbinding of RBD. Finally, an energetics analysis shows that MAN9 glycans on ACE2 decrease RBD-ACE2 affinity, while FA2 glycans lead to enhanced binding of the complex. Together, our results provide a more comprehensive picture of the detailed interplay between virus and human receptor, which is much needed for the discovery of effective treatments that aim at modulating the physical-chemical properties of this virus.