We determined the direction and strength of the photospheric and lower chromospheric magnetic field in the umbra and penumbra of a sunspot from inversions of spectropolarimetric observations of photospheric lines at 617 nm and 1565 nm and the chromospheric Ca ii IR line at 854 nm, respectively. We compare the magnetic field vector with the direction of 75 flow channels that harbor the chromospheric inverse Evershed effect (IEF) near their downflow points (DFPs) in the sunspot's penumbra. The azimuth and inclination of the IEF channels to the line of sight (LOS) were derived from spatial maps of the LOS velocity and line-core intensity of the Ca ii IR line and a thermal inversion of the Ca ii IR spectra to obtain temperature cubes. We find that the flow direction of the IEF near the DFPs is aligned with the photospheric magnetic field to within about 15°. The IEF flow fibrils make an angle of 30°-90° to the local vertical with an average value of about 65°. The average field strength at the DFPs is about 1.3 kG. Our findings suggest that the IEF in the lower chromosphere is a field-aligned siphon flow, where the larger field strength at the inner footpoints together with the lower temperature in the penumbra causes the necessary gas pressure difference relative to the outer footpoints in the hotter quiet Sun with lower magnetic field strength. The IEF connects to magnetic field lines that are not, like in the case of the regular Evershed flow, but which continue upward into the chromosphere, indicating an "uncombed" penumbral structure.