We have obtained Gemini Planet Imager (GPI) J-, H-, K1-, and K2-Spec observations of the iconic debris ring around the young, main-sequence star HR 4796A. We applied several point-spread function (PSF) subtraction techniques to the observations (Mask-and-Interpolate, RDI-NMF, RDI-KLIP, and ADI-KLIP) to measure the geometric parameters and the scattering phase function for the disk. To understand the systematic errors associated with PSF subtraction, we also forward-modeled the observations using a Markov Chain Monte Carlo framework and a simple model for the disk. We found that measurements of the disk geometric parameters were robust, with all of our analyses yielding consistent results; however, measurements of the scattering phase function were challenging to reconstruct from PSF-subtracted images, despite extensive testing. As a result, we estimated the scattering phase function using disk modeling. We searched for a dependence of the scattering phase function with respect to the GPI filters but found none. We compared the H-band scattering phase function with that measured by Hubble Space Telescope STIS at visual wavelengths and discovered a blue color at small scattering angles and a red color at large scattering angles, consistent with predictions and laboratory measurements of large grains. Finally, we successfully modeled the SPHERE H2 HR 4796A scattered phase function using a distribution of hollow spheres composed of silicates, carbon, and metallic iron.