High-elasticity anti-rutting additives, one of the most promising modifiers of asphalt mixtures, were commonly utilized for mixtures by mixing with aggregates directly. However, their interaction with the asphalt binder and their modification mechanism remain unclear. Accordingly, to understand the rheological behavior of asphalt binder modified with high-elasticity anti-rutting additives, three additives were added into asphalt binder to prepare high-elasticity modified asphalts (HEMAs) via a high-speed shear mixer, and the typical attributes of viscoelastic properties were investigated by the rotational viscosity (RV) test, frequency sweep test, as well as bending beam rheometer test. Following that, the relation between the macro-rheological behavior and microscopic mechanism was incorporated by applying Fourier transform infrared spectroscopy (FTIR), atomic force microscope (AFM), and fluorescence microscopy (FM) analysis, respectively. Results indicate that the incorporation of high-elasticity additives into base asphalt causes a remarkable increase in RV along with a decline in temperature susceptibility. Likewise, a higher proportion of elasticity behavior is indicated for HEMAs, for which the wider temperature range of elasticity dominance also occurs. Regarding modification mechanism, the interaction between the asphalt and additives is physical co-blending, without new characteristic peaks appearing in FTIR spectrum. The AFM test reveals the grouping effect of “bee-structures” conduces the agglomeration of wax crystals and macromolecular components, thus increasing the elasticity and RV of HEMAs. However, the poor compatibility between the ARA-U additive and asphalt leads to unsatisfactory low-temperature performance. Significantly, a large area percentage of “sea-island structure” implies better compatibility of ARA-N, which strongly confirms the existence of the plateau region in master curve. It recommends employing anti-rutting additives that are compatible with the asphalt and enable moderately increase its elasticity, which ensure considerable entanglement and sufficient network in the asphalt-additive system, so that balanced performance can be achieved for the asphalt binder.