Rapid and stepwise changes of the magnetic field are often observed during flares but cannot be explained by models yet. Using a 45 minute sequence of Solar Dynamics Observatory/Helioseismic and Magnetic Imager 135 s fast-cadence vector magnetograms of the X1 flare on 2014 March 29 we construct, at each timestep, nonlinear force-free models for the coronal magnetic field. Observed flare-related changes in the line-of-sight magnetic field BLOS at the photosphere and chromosphere are compared with changes in the magnetic fields in the models. We find a moderate agreement at the photospheric layer (the basis for the models), but no agreement at chromospheric layers. The observed changes at the photosphere and chromosphere are surprisingly different, and are unlikely to be reproduced by a force-free model. The observed changes are likely to require a change in the magnitude of the field, not just in its direction.