The effect of alloying elements on grain boundary sliding was systematically investigated using several binary magnesium alloys (X = Ag, Al, Li, Sn, Pb, Y and Zn) via both experimental and numerical methods. The alloying element clearly affected damping properties related to grain boundary sliding, as measured by nanoindentation tests. The properties, such as damping capacity and strain rate sensitivity, apparently depended on grain boundary characteristics, i.e., the grain boundary energy. By increasing and decreasing the grain boundary energy, the alloying element was found to play a role in enhancing and suppressing grain boundary sliding, respectively. First-principles calculations revealed that the lithium element had weak bonding to magnesium due to a few operations of the electric orbit. On the other hand, rare-earth elements exhibited relatively strong bonding to magnesium, because of electron interactions with the first nearest neighbor site, and tended to prevent grain boundary sliding. These results suggest that grain boundary energy is a reliable parameter for predicting grain boundary sliding and developing a magnesium alloy, which has good elongation-to-failure and/or secondary formability at room temperature.