We investigate the impact of asymmetric fermionic dark matter (DM) on the thermal evolution of neutron stars (NSs), considering a scenario where DM interacts with baryonic matter (BM) through gravity. Employing the two-fluid formalism, our analysis reveals that DM accrued within the NS core exerts an inward gravitational pull on the outer layers composed of BM. This gravitational interaction results in a noticeable increase in baryonic density within the core of the NS. Consequently, it strongly affects the star's thermal evolution by triggering an early onset of the direct Urca (DU) processes, causing an enhanced neutrino emission and rapid star cooling. Moreover, the photon emission from the star's surface is modified due to a reduction of radius. We demonstrate the effect of DM gravitational pull on nucleonic and hyperonic DU processes that become kinematically allowed even for NSs of low mass. We then discuss the significance of observing NSs at various distances from the Galactic center. Given that the DM distribution peaks toward the Galactic center, NSs within this central region are expected to harbor higher fractions of DM, potentially leading to distinct cooling behaviors.