Crystalline solids typically contain large amounts of defects such as dislocations and interstitials. How they travel across grain boundaries (GBs) under external stress is crucial to understand the mechanical properties of polycrystalline materials. Here, we experimentally and theoretically investigate with single-particle resolution how the atomic structure of GBs affects the dynamics of interstitial defects driven across monolayer colloidal polycrystals. Owing to the complex inherent GB structure, we observe a rich dynamical behavior of defects near GBs. Below a critical driving force defects cannot cross GBs, resulting in their accumulation near these locations. Under certain conditions, defects are reflected at GBs, leading to their enrichment at specific regions within polycrystals. The channeling of defects within samples of specifically-designed GB structures opens up the possibility to design novel materials that are able to confine the spread of damage to certain regions.