A handful of active galactic nuclei (AGN) have shown strong spectral variations in the optical band between epochs that are years apart. The appearance or disappearance of broad emission lines in their spectra completely changes their classification. Since their nucleus orientation cannot change on such short timescales, another physical interpretation has to be found. Several scenarios have been offered to explain their changing-look nature, and for the first time, we conduct polarized radiative transfer Monte Carlo simulations for all the models. We demonstrate that all interpretations have distinctive features in both total optical flux and continuum polarization, such as suggested by pioneering previous observations. Distinguishing between the different scenarios is thus straightforward. We apply our results on the changing-look quasar J1011+5442 and confirm the conclusions found previously: in this specific case, the disappearance of the broad emission lines is due to a change in accretion rate.