How the accreted mass settling on the surface of a neutron star affects the topology of the magnetic field and how the secular evolution of the binary system depends on the magnetic field change is still an open issue. We report evidence for a clear drop in the observed magnetic field in the accreting pulsar V0332+53 after undergoing a bright 3-month long X-ray outburst. We determine the field from the position of the fundamental cyclotron line in its X-ray spectrum and relate it to the luminosity. For equal levels of luminosity, in the declining phase we measure a systematically lower value of the cyclotron line energy with respect to the rising phase. This results in a drop of ∼1.7 × 1011 G of the observed field between the onset and the end of the outburst. The settling of the accreted plasma on to the polar cap seems to induce a distortion of the magnetic field lines weakening their intensity along the accretion columns. Therefore, the dissipation rate of the magnetic field could be much faster than previously estimated, unless the field is able to restore its original configuration on a time-scale comparable with the outbursts recurrence time.