We present a new method for neutrino-matter coupling in multi-dimensional radiation-hydrodynamic simulations of core-collapse supernovae (CCSNe) with the full Boltzmann neutrino transport. This development is motivated by the fact that accurate conservation of momentum is required for reliable numerical modelings of CCSN dynamics including a recoil of proto-neutron stars (PNSs). The new method is built on a hybrid approach in which we use the energy-momentum tensor of neutrinos to compute the momentum feedback from neutrino to matter in the optically thick region while we employ the collision term in the optically thin region. In this method we utilize a general relativistic description of radiation-hydrodynamics with angular moments, which allows us to evaluate the momentum feedback from neutrino to matter without inconsistency with our Boltzmann solver. We demonstrate that the new method substantially improves the accuracy of linear momentum conservation in our CCSN simulations under reasonable angular resolutions in momentum space, alleviating the difficulty in giving the diffusion limit precisely with the discrete ordinate (Sn) method. It is the first ever demonstration that the PNS kick can be handled directly and properly in multi-dimensional radiation-hydrodynamic simulations with the full Boltzmann neutrino transport.